installing Ubuntu https PPA's with squid caching

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

installing Ubuntu https PPA's with squid caching

Stuart McGraw
TL;DR: How to set up apt and a squid proxy so that
https urls are cached?

I recently tried a novel (to me) way of installing
and configuring Ubuntu by scripting the install.
The idea is that the script(s) provide a record of
what and how things were installed, provide for
disaster recovery (I backup only user files), allow
for easier future reinstalls, and I can duplicate my
current configuration in a VM for testing new
software without risk of trashing my main machine.

Getting the scripts working (and maintaining changes
going forward) requires running them dozens of times.

But... I live in third-world America and have a slow
internet connection with a data cap.

The single thing that makes it practical to do was
setting up a Squid caching proxy on another local
machine and configuring Apt to use it when installing
in order not to download GBs of packages multiple
times.

The problem I am finding is that as I add new
software many PPA repos use https rather than
http for access and Squid doesn't seem cache these
packages.  I am also concerned that Ubuntu will
at some point switch to https which will, for me,
kill any possibility of using scripts.

Is there some way of setting up Apt and the squid
proxy so that it will cache https urls?

I currently set the following on the machine being
installed:
   # cat /etc/apt/apt.conf.d/20proxy
   Acquire::http::Proxy "http://srvr1.home:3128/";
   Acquire::https::Proxy "http://srvr1.home:3128/";

Googling has turned up a lot of info that was old,
not concerned with caching, expected to work with
every app and every user (I care only about apt
and root), complex/over-generalized (SSLbump), etc.

Does anyone have any suggestions for my particular
use case?

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: installing Ubuntu https PPA's with squid caching

Colin Watson
On Sun, Nov 18, 2018 at 10:16:48PM -0700, Stuart McGraw wrote:
> TL;DR: How to set up apt and a squid proxy so that
> https urls are cached?

The standard mechanism for proxying HTTPS, and as far as I know the only
method that apt supports directly for https:// URLs, asks the proxy to
set up a tunnelled encrypted connection (using the HTTP CONNECT verb)
and then does end-to-end-encrypted communication with the target host.
When using this setup, squid can really only shuffle bytes back and
forward; it doesn't see enough of the request or response to be able to
cache anything.  (Of course this is still sometimes useful in situations
where the issue is connectivity rather than bandwidth.)

I think your best bet is going to be apt-cacher-ng: you can use its
"remapping" facility to tell it that a given set of http:// and/or
https:// mirrors are in fact to be considered identical for the purposes
of caching, or it has a hack where you can put "HTTPS///" in the URL so
that apt thinks it's using HTTP but then apt-cacher-ng uses HTTPS to
talk to the upstream mirror.  Either way is going to involve adjusting
URLs on the client side, but it sounds like that will be worth it for
you.

Some links:

  https://www.unix-ag.uni-kl.de/~bloch/acng/html/config-servquick.html#config-client
  https://www.unix-ag.uni-kl.de/~bloch/acng/html/config-serv.html#repmap
  https://www.unix-ag.uni-kl.de/~bloch/acng/html/howtos.html#ssluse

--
Colin Watson                                       [[hidden email]]

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: installing Ubuntu https PPA's with squid caching

Stuart McGraw
On 11/20/18 1:24 AM, Colin Watson wrote:

> On Sun, Nov 18, 2018 at 10:16:48PM -0700, Stuart McGraw wrote:
>> TL;DR: How to set up apt and a squid proxy so that
>> https urls are cached?
>
> The standard mechanism for proxying HTTPS, and as far as I know the only
> method that apt supports directly for https:// URLs, asks the proxy to
> set up a tunnelled encrypted connection (using the HTTP CONNECT verb)
> and then does end-to-end-encrypted communication with the target host.
> When using this setup, squid can really only shuffle bytes back and
> forward; it doesn't see enough of the request or response to be able to
> cache anything.  (Of course this is still sometimes useful in situations
> where the issue is connectivity rather than bandwidth.)
>
> I think your best bet is going to be apt-cacher-ng: you can use its
> "remapping" facility to tell it that a given set of http:// and/or
> https:// mirrors are in fact to be considered identical for the purposes
> of caching, or it has a hack where you can put "HTTPS///" in the URL so
> that apt thinks it's using HTTP but then apt-cacher-ng uses HTTPS to
> talk to the upstream mirror.  Either way is going to involve adjusting
> URLs on the client side, but it sounds like that will be worth it for
> you.
>
> Some links:
>
>    https://www.unix-ag.uni-kl.de/~bloch/acng/html/config-servquick.html#config-client
>    https://www.unix-ag.uni-kl.de/~bloch/acng/html/config-serv.html#repmap
>    https://www.unix-ag.uni-kl.de/~bloch/acng/html/howtos.html#ssluse
>

Thanks, I've started looking at acng but it feels like a sub-optimal
solution:
- Project hasn't seen any activity in 5 years
- Caching seems to be a complex business and seems like Squid or some
   thing similar is likely more widely used and hence more robust.
- acng is limited to Ubuntu/Debian servers (my current Squid solution
   is running on an old Fedora box and I hoped to also cache Fedora rpms.)
However, if that is the only viable option then that's just how things
are.  :-(

I guess I remain surprised at the lack of a good general solution since
it seems this is a growing problem with the promotion of https everywhere.
I had hoped that Squid could accept an http connection from a client and
proxy it as a https connection to the destination server (glossing over
how it knows whether to do that or not).  But since I have only rudimentary
knowledge of modern http and https, I am probably missing some technical
difficulties.

Thanks for the info.


--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: installing Ubuntu https PPA's with squid caching

Colin Watson
On Wed, Nov 21, 2018 at 02:26:40PM -0700, Stuart McGraw wrote:
> Thanks, I've started looking at acng but it feels like a sub-optimal
> solution:
> - Project hasn't seen any activity in 5 years

This seems untrue (even if the upstream web page hasn't been touched - I
don't know).  See https://tracker.debian.org/pkg/apt-cacher-ng

> - Caching seems to be a complex business and seems like Squid or some
>   thing similar is likely more widely used and hence more robust.
> - acng is limited to Ubuntu/Debian servers (my current Squid solution
>   is running on an old Fedora box and I hoped to also cache Fedora rpms.)
> However, if that is the only viable option then that's just how things
> are.  :-(

Indeed.

> I guess I remain surprised at the lack of a good general solution since
> it seems this is a growing problem with the promotion of https everywhere.
> I had hoped that Squid could accept an http connection from a client and
> proxy it as a https connection to the destination server (glossing over
> how it knows whether to do that or not).

I think the problem is more persuading the client to do that; if all
squid sees is a CONNECT request there's not much it can do about it.

--
Colin Watson                                       [[hidden email]]

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: installing Ubuntu https PPA's with squid caching

Stuart McGraw
Sorry for the delayed reply, I was away for the holidays.

On 11/22/18 5:29 AM, Colin Watson wrote:
> On Wed, Nov 21, 2018 at 02:26:40PM -0700, Stuart McGraw wrote:
>> Thanks, I've started looking at acng but it feels like a sub-optimal
>> solution:
>> - Project hasn't seen any activity in 5 years
>
> This seems untrue (even if the upstream web page hasn't been touched - I
> don't know).  See https://tracker.debian.org/pkg/apt-cacher-ng

Oops, you are correct.  I was somehow looking at a github clone I
erroneously thought was the upstream source.

>> - Caching seems to be a complex business and seems like Squid or some
>>    thing similar is likely more widely used and hence more robust.
>> - acng is limited to Ubuntu/Debian servers (my current Squid solution
>>    is running on an old Fedora box and I hoped to also cache Fedora rpms.)
>> However, if that is the only viable option then that's just how things
>> are.  :-(
>
> Indeed.
>
>> I guess I remain surprised at the lack of a good general solution since
>> it seems this is a growing problem with the promotion of https everywhere.
>> I had hoped that Squid could accept an http connection from a client and
>> proxy it as a https connection to the destination server (glossing over
>> how it knows whether to do that or not).
>
> I think the problem is more persuading the client to do that; if all
> squid sees is a CONNECT request there's not much it can do about it.

I was envisioning the client (apt in this case) connecting to the
proxy (squid) via http (isn't that the way it was done before CONNECT
existed?) and the proxy would talk to the destination server via https.
I was hoping there were some secret options in apt and/or squid to allow
this to be done.  It seems there is not.  :-(

I'm in the process now of trying to use squid's "ssl bump" feature to
decrypt apt's https requests and cache them but it seems like a lot
of complexity and fragility for what seems like a common use case.

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
Reply | Threaded
Open this post in threaded view
|

Re: installing Ubuntu https PPA's with squid caching

Hylton Conacher (ZR1HPC)
In reply to this post by Stuart McGraw
Hi Stuart,

Sri for coming in late.

On 2018/11/19 07:16, Stuart McGraw wrote:

> TL;DR: How to set up apt and a squid proxy so that
> https urls are cached?
>
> I recently tried a novel (to me) way of installing
> and configuring Ubuntu by scripting the install.
> The idea is that the script(s) provide a record of
> what and how things were installed, provide for
> disaster recovery (I backup only user files), allow
> for easier future reinstalls, and I can duplicate my
> current configuration in a VM for testing new
> software without risk of trashing my main machine.
>
> Getting the scripts working (and maintaining changes
> going forward) requires running them dozens of times.
>
> But... I live in third-world America and have a slow
> internet connection with a data cap.
>
> The single thing that makes it practical to do was
> setting up a Squid caching proxy on another local
> machine and configuring Apt to use it when installing
> in order not to download GBs of packages multiple
> times.
>
> The problem I am finding is that as I add new
> software many PPA repos use https rather than
> http for access and Squid doesn't seem cache these
> packages.  I am also concerned that Ubuntu will
> at some point switch to https which will, for me,
> kill any possibility of using scripts.
>
> Is there some way of setting up Apt and the squid
> proxy so that it will cache https urls?


Ouch, I think we all have experienced slow Internet and data caps. Why
not set-up your other machine as a firewall <https://www.ipfire.org>
with an included squid caching proxy with a static 'green interface' IP?
I know I used to use its predecessor IPCop for the same but dev work has
stopped on IPCop.

Then it does not matter how many times you request a file from the net,
its going to check the proxy first.

Just a thought.

Regards
Hylton

--
ubuntu-users mailing list
[hidden email]
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-users