1. Bandwidth
2. Latency

In an ideal world, the latency is low and bandwidth is high.

Bandwidth problems become more important when the things being sent over the network are large files ie: >1MB. Latency issues become more important when the things being sent become smaller/require realtime feedback ie: < 100kb.

Reducing page size & The time to first byte can be critical if you want to target users on high latency, low bandwidth environments.

The Jevons Paradox: Faster sites report slower loads

Back in 2008, the youtube website was around 1MB. The developers weren’t happy with that and created The Youtube Feather Project to reduce the page size. The prototype with limited features was made in 3 days and weighed around 98KB.

They did an A/B test with a few users for 1 week. After collecting and analyzing the data the developers were confused. The page latency increased under Feather.

The reason? Geography.

When we plotted the data geographically and compared it to our total numbers broken out by region, there was a disproportionate increase in traffic from places like Southeast Asia, South America, Africa, and even remote regions of Siberia. Further investigation revealed that, in these places, the average page load time under Feather was over TWO MINUTES! This meant that a regular video page, at over a megabyte, was taking more than TWENTY MINUTES to load! This was the penalty incurred before the video stream even had a chance to show the first frame. Correspondingly, entire populations of people simply could not use YouTube because it took too long to see anything. Under Feather, despite it taking over two minutes to get to the first frame of video, watching a video actually became a real possibility. Over the week, word of Feather had spread in these areas and our numbers were completely skewed as a result.

You can find the full post at Page Weight Matters

The irony of making something faster but analytics reports slower loads is very interesting. The general term for it is the Jevons Paradox and it applies to many things eg: Website loading speed, electricity consumption etc.

I also found it interesting that Youtube Feather and SPDY (the predecessor to http2) were announced at around the same time.

I guess this is why milisecond improvements in page loads can increase revenue.

Reducing Bandwidth

Lossless compression

Lossless compression is great for text based content and is the obvious low hanging fruit when looking at reducing bandwidth.

Here’s a chart showing how brotli (a compression format) was used to improve load speed and save bandwidth.

Addy Osmani @addyosmani

· Oct 28 2017

@slightlylate It's been awesome seeing more businesses getting compression wins with Brotli over the last year. Google's Display Ads now uses Brotli too.

6:52 PM · Oct 28, 2017

78 Link to Tweet

Notice how the Indian linkedin users get 2% faster loads out of brotli than the USA users.

Here’s another example with tailwindcss.

Adam Wathan @adamwathan

· Jul 31 2020

Still completely blows my mind how well Tailwind compresses with Brotli. Bravo to those geniuses 👏

12:27 AM · Jul 31, 2020

352 Link to Tweet

Delta Compression

Delta compression is very useful when compressing a lot of similar data. If you have a one line update to a file, you’d only need to download/store the one line change, regardless of the original file size. This is very useful when doing a lot of incremental changes.

The most common use of it is probably git. Another great example is the manyclangs project which uses elfshaker to compress 2,000 clang builds from 400 GiB down to 100 MiB and only takes 4 seconds to load any build.

Delta compression is also very useful when storing backups. You can use the bup tool, if you want to compress lots of simmilar files eg: excel sheets but don’t want to directly use git.

I’d love if chromium had something like this so i can run any chrome version I want.

Shared Dictionaries

Dictionaries are commonly used to increase compression ratios for specific loads. A dictionary contains things that appear frequently and take lots of space.

The web platform will soon add support for compression dictionaries

• Compression dictionary transport

Brotli has a feature where you can compress a new file using an old one as a dictionary and can improve compression ratios dramatically.

I did some tests and they can be really good.

Compressing typescript version 5.0.2 with version 5.1.0-beta as a shared dictionary

brotli: 642KB brotli+dict: 135KB

This is good news for sites that make constant updates.

Patrick Meenan @patmeenan

· Mar 9 2023

It's mind-blowing how well brotli shared dictionaries can help with re-delivering "mostly the same" content (both code/CSS bundles and MPA HTML). A couple of quick spoilers below...

4 12 61

Patrick Meenan @patmeenan

· Mar 9 2023

Youtube's desktop player bundle (at 10MB uncompressed) - 1.8 MB using brotli. - 384 KB upgrading from a 2-month-old version using the old version as a shared dictionary (78% smaller than brotli).

2 3 23

Patrick Meenan @patmeenan

· Mar 9 2023

The HTML for the listing pages for @eBay as you go from item to item (600k uncompressed): - 80 KB using brotli. - 22 KB using a custom dictionary built off of other listing pages (73% smaller)

2 2 11

You can perform test on your data at test.patrickmeenan.com/shared-brotli/static/

Check out the chrome status page & bug tracker for

implementation status of custom dictionaries.

Here’s the script I used

#!/bin/bash
rm ./*.br
rm ./*.js

dict="cdnjs.cloudflare.com/ajax/libs/typescript/5.0.2/typescript.min.js"
new="cdnjs.cloudflare.com/ajax/libs/typescript/5.1.0-beta/typescript.min.js"
dec="smol.s"

# Compress using File as dictionary
echo "encoding shared br"
brotli -q 5 $new -o "$dec.shared.br" -D $dict
echo "encoding basic br"
brotli -q 5 $new -o "$dec.basic.br"

# verify
echo "decoding basic br"
brotli -d "$dec.basic.br" -o "basic-decoded.js"

echo "decoding shared br"
brotli -d "$dec.shared.br" -D $dict -o "shared-decoded.js"

echo "decoding shared br"
sha256sum $new "shared-decoded.js" "basic-decoded.js"
exa --no-user --no-time --no-permissions -l ./*.br

Zstandard

Zstandard (zstd) is a compresion format from facebook that is simmilar to brotli but requires less cpu for comparable compression ratios to brotli. It’s sadly not implemented in browsers. Here’s a chart comparing them

Jyrki Alakuijala 🇺🇦 @jyzg

· Feb 9 2023

brotli/zstd/gzip compression comparison: benchmarked with silesia corpus files split into 1 MB chunks which are then compressed independently brotli used to be faster in compression at lower qualities, too, but zstd has caught up and exceeded at qualities below 5

3 4 21

音風景の管理人 @kamedo2

· Feb 9 2023

@jyzg So, brotli has the 'compression-very-slow-but-ultra-compressed' option that zstd doesn't offer, right? I have a beefy machine to compress, but my clients can be anything, including low-end smartphones and single-board computers, so it helps.

1 1

Jyrki Alakuijala 🇺🇦 @jyzg

· Feb 9 2023

@kamedo2 Brotli as a format is a bit more expressive and can give better density when the heuristics are right or thoroughly explored. Zstd as a format is easier to decompress. Brotli hides less data than Zstd -- you can decode byte-to-byte whereas Zstd only when you have a full block.

1 3

Audio Compression

There isn’t that much innovation happening in the audio compression but I found one interesting thing.

Facebook is Using AI for lossy audio compression & acheives a 10x size reduction. ie: great audio quality at smaller sizes

Here’s a chart showing the difference.

And Here’s a video demonstrating the quality differences

Video Compression

Youtube uses their own custom chips for video compression. Warehouse-scale video acceleration: co-design and deployment in the wild

The Alliance for Open Media, which is behind the av1 video codec & the avif image format

• IP Multicast - To reduce server load for broadcast streams eg: Sport Events, National events, etc

Image compression

JPEG-XL is a new universal image format that’s good at everything.

Jon Sneyers @jonsneyers

· Jan 5 2021

Lossless image compression: besides the very slow experimental compressors LEA and EMMA, #JPEGXL beats anything else by a comfortable margin. Well-optimized PNG is 33% to 66% larger than #JXL . Not so well-optimized PNGs in the wild (not shown here) are ~50% to 100% larger.

9 41 146

Jon Sneyers @jonsneyers

· Jan 5 2021

Also in terms of speed, JXL has a nice range of trade-offs between density and encode time.

1 1 11

Jon Sneyers @jonsneyers

· Jan 6 2021

I didn't do these benchmarks myself, I just summarized the results. The data is coming from docs.google.com/spreadsheets/d…, which I saw in this forum thread: encode.su/threads/3544-J…

docs.google.com

Lossless Image Formats Comparison (Jpeg XL, AVIF, WebP 2, WebP, FLIF, PNG, ...) v1.27

Details Summary,Size difference vs. JPEG XL (-e 9 -E 3 -I 1) Type,Image category,WebP,WebP v2,FLIF,AVIF,PNG,Effort 9,Effort 7,Effort 3,BMF (-s),EMMA Digital 2d Art,Art,16.8%,10.2%,5.8%,71.3%,47.7%,...

1 4

https://encode.su/threads/3544-JXL-reference-software-version-0-2-released?p=67989&viewfull=1#post67989

Chart is from encode.su

Latency

The other problem to solve when dealing with web performance is latency, especially network latency. I really like the post titled: It’s the Latency, Stupid which goes over why latency is important and why it’s hard to reduce it compared to increasing bandwidth.

There’s several ways to reduce latency as a developer.

Do all the processing locally.

Eliminating the server and doing everything on the client side is a common way to tackle performance problems.

A good example is squoosh.app an image compression web app. It uses webAssembly which is slower than native but overall much faster than using an online image compression service for one-off transformations.

Latency is the main reason Single Page Apps are a thing and why client side routing is always used for long running web applications.

Evan You @youyuxi

· Apr 12 2022

I feel that the current web perf discourse obsesses too much on initial loading perf and largely ignores subsequent navigations. It’s not even quantified in lighthouse scores, but makes a noticeable difference even for static sites.

Ward Peeters @wardpeet

· Apr 11 2022

SPA navigations always feel faster than MPA to me. Even turbo doesn't feel amazing. Anyone has an example to change my mind?

10:33 AM · Apr 12, 2022

709 Link to Tweet

There’s one issue with client side routing, it’s fragile and requires a lot of tooling and monitoring. Although that’s an implementation detail not tied to all SPAs. There’s SPA frameworks like elm with zero runtime exceptions.

Here’s a video from 2016 showing a bug (hopefully fixed?) when using github from airport wifi. Opening a new tab was faster than the client router.

source: Youtube

SPAs need more work to function properly compared to simple MPAs, especially for document based sites. Doing a lot of things in the client can lead to more crashes that defeat the whole point of improving user experience.

pre-fetch things the application will need

When visiting some documentation websites like kit.svelte.dev, devdocs.io/ from a desktop device, the page navigation is much faster than anything a multi page app could ever do.

Pre fetching things is not always good and could have a negative impact on users on low-end devices.

Use modern protocols

Specifically, adjust the backend to support batch requests and enable modern protocols like http2 and http3.

It’s pretty easy to setup a http2 server but surprisingly hard to make http2 requests from some languages. The solutions I could find that required little configuration were the command line tools curl & wget2 and the python httpx library.

I liked the python one since it allowed me to reuse a client in 3 lines of code despite me not really writing that much python code

Here’s an example

client = httpx.AsyncClient(http2=True)
response = await client.get(...)
print(response.http_version)
# "HTTP/1.0", "HTTP/1.1", or "HTTP/2".

I noticed the benefit of http2 when I had to making more than 10,000 http requests to the twitter api for my archive-browser application. I did some tests using curl and the difference is big

Move the servers closer to the user for dynamic applications eg: game servers CDNs

Improve the hardware

Build a own private fiber optic cable network eg: Cloudflare Backbone and directly lower the latency

Measurments & analytics

All these are great strategies but need to be measurably faster espacially if you have more users. Here’s a story about how netflix preloading data lead to increased crashes & less engagement, which is bad.

Source: Clean Code : Horrible Performance on youtube

The important takeaway here is to measure and test everything and challenge assumptions. Not that doing things in the client is bad.

You are not Google

All these tips are only important if you have a lot of users on restricted devices. If you have few users on predictable hardware and your app works, performance isn’t really that important

Further reading

Here’s some relevant links

• Brotli compression using a reduced dictionary
• Improving compression with a preset DEFLATE dictionary
• The Website Obesity Crisis
• The modern web on a slow connection
• web-really-sucks-slow-connection
• How the BBC and ITV are fixing delays on World Cup live streams

More topics

I haven’t covered everything performance related. Here’s some of the topics.

• http2 & http3 multiplexing
• TLS 1.3 0 RTT
• Lighthouse
• Profiling websites with v8 profiler
• CSS Performance (is it even a thing)
• Image codec cpu usage
• Geographical Distribution
• Zstandard custom dictionaries
• Brotli cpu usage benchmarks vs other codecs for compression on the fly
• Image compression services
• Service Workers

"What is Cumulative Layout Shift (CLS) and why does it matter?" This is probably my favorite example.

MidlineShift @detroit_bound

· Mar 27 2022

@ampol_moment

3:13 AM · Apr 13, 2022

1.1K Link to Tweet

Preventing CLS requires setting explicit width and height attributes on all pictures. Here are the steps I took to optimize the images in hugo using the render hooks feature.

1. Get the raw image data from a URL
2. Decode the image data
3. If available, set the width & height attributes
4. Otherwise, fallback to the default template without the attributes

Here’s the complete code snippet.

<!-- layouts/_default/_markup/render-image.html -->
{{ $alt := .Text }}
{{ $src := .Destination }}
<!-- Load & decode image -->
{{ $img := os.ReadFile $src | resources.FromString $src }}

{{/* Set Dimensions on Raster Images */}}
{{ if ne $img.MediaType.SubType "svg" }}
  <img
    alt="{{ $alt }}"
    src="{{ $src | safeURL }}"
    width="{{ $img.Width }}"
    height="{{ $img.Height }}"
/>
{{ end }}

{{/* Svgs & Images that failed to load */}}
{{ if or (not $img) (eq $img.MediaType.SubType "svg") }}
  <img src="{{ $src | safeURL }}" alt="{{ $alt }}" />
{{ end }}

For this render hook to work, images must be stored in the /img folder relative to the project root with this configuration file.

# config.yaml
module:
  mounts:
    - source: static
      target: static
    - source: img
      target: static/img

Further development

• Suport Remote Image sources.
• Support different local image sources eg: /content, /static or /assets.
• Multiple image formats & sizes.

invalid character '/' looking for beginning of object key string

To prevent this error, you need to sanitize the json using the hujson package from Tailscale.

Here’s an example on how to use it, also available in the Go playground.

 1package main
 2
 3import (
 4	"encoding/json"
 5	"fmt"
 6
 7	"github.com/tailscale/hujson"
 8)
 9
10const data = `{
11  // Comment
12  "foo": "bar",
13}`
14
15type x struct {
16	Foo string `json:"foo"`
17}
18
19func main() {
20	var t x
21	data, err := standardizeJSON([]byte(data))
22	err = json.Unmarshal(data, &t)
23	if err != nil {
24		fmt.Println(err)
25		return
26	}
27
28	fmt.Println(t.Foo)
29}
30
31func standardizeJSON(b []byte) ([]byte, error) {
32	ast, err := hujson.Parse(b)
33	if err != nil {
34		return b, err
35	}
36	ast.Standardize()
37	return ast.Pack(), nil
38}

Here’s an example of the code twitter provides.

<!-- Markup-->
<blockquote class="twitter-tweet">
  <p>just setting up my twttr</p>
  — jack⚡️ (@jack) March 21, 2006
</blockquote>
<!--Script to enhance markup-->
<script src="https://platform.twitter.com/widgets.js"></script>

The problem

This method of embedding tweets has some disadvantages:

• Causes a lot of layout shifting
• Difficult to style
• Does not preserve deleted tweets

The solution

A good solution would be to get the tweet data from the API then use plain HTML & CSS to display it, which gives you the best lighthouse scores.

I implemented one for this blog. Here’s an example.

{{<tweet id="20">}}

jack @jack

· Mar 21 2006

just setting up my twttr

8:50 PM · Mar 21, 2006

184.3K Link to Tweet

Biz Stone @biz

· Mar 21 2006

wishing I had another sammich

9:16 PM · Mar 21, 2006

3.6K Link to Tweet

I had a lot of fun implementing it. I even added some features not available in the official embed.

Features

Quote tweets

Evan You @youyuxi

· Apr 12 2022

I feel that the current web perf discourse obsesses too much on initial loading perf and largely ignores subsequent navigations. It’s not even quantified in lighthouse scores, but makes a noticeable difference even for static sites.

Ward Peeters @wardpeet

· Apr 11 2022

SPA navigations always feel faster than MPA to me. Even turbo doesn't feel amazing. Anyone has an example to change my mind?

10:33 AM · Apr 12, 2022

709 Link to Tweet

Threads

Twitter API @TwitterAPI

· Apr 29 2022

Today, we’re deprecating two features of the legacy v1.1 streaming endpoints, which we will retire on Oct 29, 2022: • The v1.1 statuses/sample endpoint • Compliance messages on the v1.1 statuses/filter endpoint Learn more: twittercommunity.com/t/deprecation-…

twittercommunity.com

Deprecation announcement: Removing compliance messages from statuses/filter and retiring statuses...

Today, we’re deprecating two features on the Twitter API v1.1 streaming endpoints: The v1.1 statuses/sample endpoint Compliance messages on the v1.1 statuses/filter endpoint These features will be...

7 28 79

Twitter API @TwitterAPI

· Apr 29 2022

The v2 sampled stream, filtered stream, and batch compliance endpoints are available to all developers as replacements for this v1.1 functionality.

4 5 12

Twitter API @TwitterAPI

· Apr 29 2022

Starting today, new client applications will not be able to access v1.1 statuses/sample nor statuses/filter. Client apps created before today will retain access to both endpoints until they are retired. We are not deprecating v1.1 statuses/filter today, only compliance messages.

3 11 33

Twitter API @TwitterAPI

· Apr 29 2022

Deprecating these legacy features enables us to best serve developers on the Twitter API v2. We regain energy to go faster on new features. We’re committed to making migration to v2 equivalents easy and building more new value on v2. See the forum post for resources 👆

2 6 33

TypeScript @typescript

· Mar 9 2022

Could we have TypeScript with no build step? Could we have types in JavaScript? Maybe! We're exploring bringing optional type syntax into the JavaScript language! devblogs.microsoft.com/typescript/a-p…

devblogs.microsoft.com

A Proposal For Type Syntax in JavaScript - TypeScript

Today we’re excited to announce our support and collaboration on a new Stage 0 proposal to bring optional and erasable type syntax to JavaScript. Because this new syntax wouldn’t change how surroun...

154 1.5K 5.6K

TypeScript @typescript

· Mar 9 2022

The Types as Comments proposal means that browsers could run TypeScript-checked code directly, (even though they wouldn't do any type-checking). That means faster iteration with all the type-checking and editing experience you know and love today.

5 18 334

TypeScript @typescript

· Mar 9 2022

We're excited to hear what you think of this idea! We're hoping that this proposal can make coding easier for TypeScript and JavaScript users at every scale.

17 8 260

OpenGraph Previews

Andrew Ayer @__agwa

· Nov 12 2021

New blog post: It's Now Possible To Sign Arbitrary Data With Your SSH Keys agwa.name/blog/post/ssh_…

agwa.name

It's Now Possible To Sign Arbitrary Data With Your SSH Keys

ssh-keygen can sign and verify signatures, and it's way better than PGP

4:19 PM · Nov 12, 2021

276 Link to Tweet

Eric Bailey @ericwbailey

· Jan 11 2022

"The real-life impact of our technical decisions really hit home to me once again: my Mom had trouble volunteering and participating in her local community because somebody shipped the optional chaining operator in their production JavaScript." blog.jim-nielsen.com/2022/a-web-for…

blog.jim-nielsen.com

The Optional Chaining Operator, “Modern” Browsers, and My Mom

5:04 PM · Jan 11, 2022

182 Link to Tweet

Cloudflare @Cloudflare

· Oct 4 2021

“Facebook can't be down, can it?”, we thought, for a second. Well it can, and here's how. cfl.re/3l7WBQs

blog.cloudflare.com

Understanding how Facebook disappeared from the Internet

Today at 1651 UTC, we opened an internal incident entitled "Facebook DNS lookup returning SERVFAIL" because we were worried that something was wrong with our DNS resolver 1.1.1.1. But as we were...

9:16 PM · Oct 4, 2021

5.1K Link to Tweet

Polls

Lea Verou @LeaVerou

· Mar 16 2022

Hey, people who write CSS! Do you use a preprocessor, and if so, which one?

No, I don’t   1532 28%

Yes, PostCSS   924 17%

Yes, Sass   2838 51%

Yes, other   275 5%

5569 votes · Final results

2:29 PM · Mar 16, 2022

66 Link to Tweet

Images

If supported, profile pictures & images are served as webp, which provides smaller sizes than jpeg and is supported in 90% of browsers

API demos @FloodSocial

· Jun 25 2021

Growing more than apples

11:58 PM · Jun 25, 2021

Link to Tweet

API demos @FloodSocial

· May 29 2017

Geo with media

10:20 PM · May 29, 2017 from the pool

1 Link to Tweet

API demos @FloodSocial

· May 8 2017

Test Tweet with @mentionThis $twtr bit.ly/2pUk4be #hashtag

5:02 PM · May 8, 2017 from Longmont, CO

8 Link to Tweet

Gifs & Videos

API demos @FloodSocial

· May 31 2017

Test Tweet with animated GIF

10:22 PM · May 31, 2017 from Boulder, CO

3 Link to Tweet

ladies and gentlemen, the weekend 😌 @CraigWeekend

· Apr 1 2022

5:52 PM · Apr 1, 2022

61.6K Link to Tweet

Matisse & Sadko @MatisseSadko

· Apr 12 2022

Good morning guys! We’re thrilled to present you a new collab with our brother @MartinGarrix 🔥 stmpd.co/vPLaYHI3

6:18 PM · Apr 12, 2022

435 Link to Tweet

Hashflags

One of the unreleased features I added were hashflags, the emojis that appear after a hashtag.

Here’s an example using an expired hashflag.

Jane Manchun Wong @wongmjane

· Jun 3 2021

Apple’s #WWDC21 hashflag campaign name is absolutely wild: USEN_THB_TLB_72x72_BAN_WWDC_PREE_NA_NA_NA_NA_NA na na na na na

8:49 AM · Jun 3, 2021

415 Link to Tweet

The code that archives deleted hashflags can be found at github.com/ianmuchina/HashflagArchive repo. It updates itself via github actions and a script.

Spaces

I also added a Twitter spaces card. It’s simple and only links to the space page on twitter.

Bryan Cantrill @bcantrill

· Apr 11 2022

No Twitter Space today, but @ahl and I will be back next week where we will be joined by @SyntheticGate, @random_enginerd, @AaronHartwig1, @arjenroodselaar and crew with more tales from the bringup lab! (The sequel to youtube.com/watch?v=lhji-k….)

Apr 18 Bryan Cantrill 2:07:57

More tales from the bringup lab

Open on Twitter

5:29 PM · Apr 11, 2022

25 Link to Tweet

After the space expires it will display like this.

Twitter Dev @TwitterDev

· Mar 23 2022

Join @ashevat, @jessicagarson, @suhemparack and @alanbenlee Thursday 3/24 at 3:05pm PT for this month’s town hall conversation on the recent updates to the Twitter Developer Platform. 📆 Space Url:

Spaces

Details not available

8:19 PM · Mar 23, 2022

18 Link to Tweet

More tests are available at

drafts.blog-byl.pages.dev/tests/tweet/

Deleted tweets

Tweets are still shown if they get deleted.

muchina @much1na

· Apr 29 2022

This tweet does not exist

10:55 PM · Apr 29, 2022

Link to Tweet

Further development

• More Test cases
• avif file format
• Threads
• Image zooming
• Image placeholders

For the static site generator, I chose Hugo because of its speed & simplicity. The tradeoff is that feature-wise, it’s limited. Although it doesn’t matter that much for a blog.

I also transferred my domain from NameCheap to Cloudflare. It made sense since I was already using Cloudflare for DNS. The costs are crazy. Literally wholesale pricing. Only on transfers & renewals. You need an existing domain.

• Cloudflare Supported Tlds
• Wholesale TLD Prices

When chrome displays the paused sign, it’s an issue with cookies. In this case, the cookies are encrypted, and chrome can’t decrypt them because it’s looking for the decryption key in the wrong keyring.

Too many keyrings

A keyring is a lot like a password manager. The only difference is that keyrings are meant for programs, and password managers are meant for humans. For security, chrome stores the key used to decrypt cookies in the keyring.

All major operating systems have some sort of keyring implementation. Windows has one, Mac has one. Linux has several. On Linux, chrome decides the keyring it will use based on the current desktop environment.

When switching desktop environments, chrome will switch keyrings without migrating the keys. This makes chrome forget the key needed to decrypt the cookies.

Stick with one

The solution is to hardcode the keyring chrome will use. You can do this with the –password-store flag, which has 3 options:

• gnome (to use GNOME Keyring)
• kwallet (to use KDE’s KWallet)
• basic (to use the plain text store. No encryption)

To set the flag, first install menu-libre.

sudo apt install menulibre

Then using menu-libre, edit chrome’s command.

/usr/bin/google-chrome-stable --password-store=kwallet5 %U

Hopefully, that works, and you can switch desktops without breaking your browser.

Further reading

Benefits of keyrings

# Default Command
CMD="youtube-dl"

# Use yt-dlp if avaiable
command -v yt-dlp >/dev/null &&
  CMD="yt-dlp" ARGS=""

# Quality Options
Q_HD="720"
Q_FHD="1080"
Q_DEFAULT="$Q_HD"

Default command

Saves video to ~/Videos/Channel/video.mp4 without video Id.

# Video
ydl() {
  "$CMD" "$ARGS" \
    --no-playlist \
    --embed-subs \
    --format "[height<=$Q_DEFAULT]" \
    --output "$HOME/Videos/%(uploader)s/%(title)s.%(ext)s" "$@"
}

720p Video

• Single video
• No playlist
• Saves to ~/Videos/Uploader/Title.mp4

# Video (720p)
yhd() {
  "$CMD" "$ARGS" \
    --no-playlist \
    --embed-subs \
    --format "[height<=$Q_HD]" \
    --output "$HOME/Videos/%(uploader)s/%(title)s.%(ext)s" "$@"
}

720p video playlist

• Download entire playlist
• Saves to ~/Videos/Uploader/Playlist/Title.mp4

ypl() {
  "$CMD" "$ARGS" \
    --yes-playlist \
    --format "[height<=$Q_DEFAULT]" \
    --output "$HOME/Videos/%(uploader)s/%(playlist)s/%(title)s.%(ext)s" "$@"
}

Audio

• Outputs to mp3
• Does not download entire playlist
• Saves to ~/Music/Uploader/Title.mp3
• Embeds Video thumbnail as cover art

# Audio
ymp3() {
  "$CMD" "$ARGS" \
    --extract-audio \
    --audio-format mp3 \
    --no-playlist \
    --embed-thumbnail \
    --format "bestaudio" \
    --output "$HOME/Music/%(uploader)s/%(title)s.%(ext)s" "$@"
}

Audio Playlist

• Saves to ~/Videos/Uploader/Playlist/Title.mp3
• Adds thumbnail as cover art

# Audio Playlist
ypl3() {
  "$CMD" "$ARGS" \
    --extract-audio \
    --audio-format mp3 \
    --yes-playlist \
    --embed-thumbnail \
    --format "bestaudio" \
    --output "$HOME/Music/%(uploader)s/%(playlist)s/%(title)s.%(ext)s" "$@"
}

The things that can make a build non-reproducible are:

• Timestamps
• Unique IDs
• Build paths
• etc

Why they matter

Reproducible builds have inherent security. They allow us to verify the source code a binary comes from. This makes detecting changes or tampering straightforward.

1. Compile the program on at least two different systems
2. Compare the checksums.
3. If they match, that’s good. If they don’t match, something is wrong.

The builds have more integrity, which benefits everyone. For security-minded people, it means a straightforward way of detecting backdoors in the build process. For open source enthusiasts, it means a clear way of detecting GPL violations. For everyone else, we get safer software.

You can find out how to acheive reproducible builds at reproducible-builds.org/docs.

Attacks on build systems

Back-doors introduced in the build process are not easy to detect. Most of the detection happens too late when the damage is already done. The attacks can have a high impact in a short period, so early detection is important.

There have been many attacks like this in the past. Some of them are:

• SUNSPOT, An attack on SolarWinds that affected lots of important people
• XcodeGhost - Attack on an IDE for Mac. Lots of iOS apps were affected. Including Angry Birds

Defending against them

Reproducible builds are the best way to defend against these kinds of attacks. Attackers lose their incentive bacause they are detected quickly and need to compromise more systems.

The disadvantages

Like everything there are downsides. Some of them are:

• Need to use a specific compiler version
• No Profile-guided optimization

What they don’t protect us from

Reproducible builds don’t protect us from malicious developers. A developer could knowingly write vulnerable code that looks like a mistake when discovered. This called underhanded code.

• Underhanded C Contest
• Underhanded Crypto Contest

In the paper titled Trusting Trust Ken Thompson asks us:

To what extent should one trust a statement that a program is free of Trojan horses? Perhaps it is more important to trust the people who wrote the software.

Who does reproducible builds?

Many open-source projects have reproducible builds to assure users of their integrity. Some of them include:

1. Bitcoin
2. Tor Browser
3. F-Droid
4. Signal
5. Telegram

You can find more at reproducible-builds

Linux distros

Around 80%-90% of the packages in Linux distributions ( Arch, Debian, OpenSUSE, NixOS, Guix) are already reproducible. You can find the exact numbers at reproducible-builds.org.

Digital signatures

Digital signatures still have their place. They are useful when verifying who a document or message comes from. They aren’t useful when verifying the source code a binary comes from. Some forms of digital signatures can get in the way of reproducible builds, as explained by telegram developers in this article.

Conclusion

I’ll conclude by quoting Mike Perry of the Tor project. He described the importance of reproducible-builds as follows:

I don’t believe that software development models based on trusting a single party can be secure against serious adversaries anymore, given the current trends in computer security.

This statement is true to this day.

Further reading

Many people have written about reproducible builds and have gone into more detail than I have in this post. Here are some of them.

reproducible-builds.org

A website with technical information on reproducible builds. It also has status updates on Linux distributions.

Reflections on Trusting Trust

A paper by Ken Thompson. He asks us what if compilers had backdoors. Would it possible to even detect & prevent such an attack?

Countering Compiler backdoors

David A. Wheeler answers the above question. He proposes a method called Diverse Double-Compiling.

The Octopus Scanner Malware

Writeup on the discovery of a supply chain attack that targeted developer’s machines.

Verifying the source code for binaries

An Lwn article on reproducible builds

In this post, I will go over how to configure automatic security updates on Debian, which are not enabled by default.

Introduction

Debian uses the apt package manager. A system update is done with the following commands:

$ apt update
$ apt upgrade

The importance of security updates

Security updates contain fixes for security vulnerabilities. They are an overlooked protection measure against many attacks.

In 2017 there was a ransomware attack named Wannacry. All the infected computers had one thing in common. They weren’t updated.

Updating a system regularly can prevent lots of attacks.

Why Automate?

Performing manual updates is repetitive and can be a huge burden to system administrators.

Setting up automatic updates

First update the package list.

$ apt update

Install these 2 packages

$ apt install unattended-upgrades apt-listchanges

Perform a quick configuration

$ sudo dpkg-reconfigure -plow unattended-upgrades

At this point, automatic upgrades have been configured and will run every 24 hours.

Automatic reboots

The next thing is to set up automatic reboots. Some updates such as those to the Linux kernel require a reboot.

 1. With a text editor open the unattended-upgrades configuration file.

$ vim /etc/apt/apt.conf.d/50unattended-upgrades

 2. Uncomment the line with Automatic-Reboot and set it to True.

Unattended-Upgrade::Automatic-Reboot "true";

 3. This step is not neccesary but makes updates predictable.

Unattended-Upgrade::Automatic-Reboot-Time "23:00";

Commented lines are those that start with two slashes // , remove the slashes to uncomment the lines.

// This line is commented
This line is not commented

Live Patching

If server uptime is important to you, a good solution is Livepatch by Cannonical, the company behind Ubuntu. It is free for up to 3 devices.

Livepatch allows you to install some critical kernel security updates without rebooting your system, by directly patching the running kernel. … It’s mainly intended for servers which are supposed to have months and years of continuous uptime without reboots

More info on kernel live patching.

Alerts on update failure

Debian is a very stable OS, however, updates can fail when using software that is not in the official repos. When that happens you will want to receive an alert.

I will be setting up email alerts in case of update failure.

Configuring unattended-upgrades to send email

1. With a text editor modify the following lines in the unattended-upgrades configuration file

$ vim /etc/apt/apt.conf.d/50unattended-upgrades

2. Uncomment and modify these lines.

Unattended-Upgrade::Mail "your@email.address";
Unattended-Upgrade::MailOnlyOnError "true";

Setting up postfix to send email

To send the email alerts you need a working email setup. For this, I will be using postfix to send email from a Gmail account

Create an app password

If you use Gmail, a good way to control access is by using app passwords. I use this method because I can revoke app passwords in case of compromise.

An App Password is a 16-digit passcode that gives a non-Google app or device permission to access your Google Account

1. To create an app password, go to the Google account security page while signed in.
2. Under Signing into Google , Select App Passwords
3. Generate a custom App password, name it to something relevant like server-x email alerts and copy it. We’ll need it later.

Install and Configure Postfix

The next thing is to install Postfix and other required packages. They are required to send emails using the app passwords we just created.

1. Install postfix, libsasl and bsd-mailx

$ apt install postfix libsasl2-modules bsd-mailx

When prompted, select Internet site.

Configure postfix

The first step is to configure postfix. Follow these steps.

1. Open the postfix configuration file.

$ vim /etc/postfix/main.cf

2. Find the line with relayhost and add this.

relayhost = [smtp.gmail.com]:587

Set up TLS

The next step is to configure TLS. It stands for Transport Layer Security This ensures the email contents and credentials are encrypted in transit.

1. Open the postfix configuration file.

$ vim /etc/postfix/main.cf

2. Append the following lines to the end of the file

# Enable SASL authentication
smtp_sasl_auth_enable = yes
# Disallow methods that allow anonymous authentication
smtp_sasl_security_options = noanonymous
# Location of sasl_passwd
smtp_sasl_password_maps = hash:/etc/postfix/sasl/sasl_passwd
# Enable STARTTLS encryption
smtp_tls_security_level = encrypt
# Location of CA certificates
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt

Set up the credentials

The next step is to configure postfix to use the Gmail app passwords we have created.

1. Open /etc/postfix/sasl/sasl_passwd

$ vim /etc/postfix/sasl/sasl_passwd

2. Add the following contents

[smtp.gmail.com]:587 yourusername@gmail.com:password

3. Generate the password database

$ postmap /etc/postfix/sasl/sasl_passwd

Secure the credentials

By default, the files containing the email credentials are readable by everyone. We can change this.

1. Modify the permissions so that the root user owns them with chown

$ sudo chown root:root /etc/postfix/sasl/sasl_passwd \
   /etc/postfix/sasl/sasl_passwd.db

2. With chmod, change the permissions so that only root can the file.

$ sudo chmod 0600 /etc/postfix/sasl/sasl_passwd \
    /etc/postfix/sasl/sasl_passwd.db

3. Restart Postfix

$ systemctl restart postfix

Test the email setup

1. Send yourself a test email from the terminal

$ echo "Test completed" | mail -s “Test From server” your_email@gmail.com

If the email is received, then congratulations. The email setup is comlete. You now have:

• Automatic daily updates
• Automatic reboots
• A feedback system incase updates fail

Image Credits

Illustrations from Stories by Freepik

Docker is a container platform that is similar to a Hypervisor like Virtualbox. Containers use less storage and RAM and are portable.

Docker can run on:

• Linux
• Windows
• Mac OS

In this article I will go over how to set up a penetration testing lab entirely in docker

It will consist of two types of containers.

1. Attacker Machine
2. Target Machines

Installation on Linux

The Docker engine is in the official repositories of most Linux distributions.

Ubuntu/Debian

Anything that uses apt to install software is Debian based.

To install docker:

$ sudo apt install docker.io

Fedora

To install on fedora:

$ sudo dnf install docker

Start the Docker service

$ sudo systemctl start docker

Arch/Manjaro

To install Arch based distros:

$ sudo pacman -Syu

Enable the loop kernel module

$ sudo tee /etc/modules-load.d/loop.conf <<< "loop"
$ modprobe loop

Install Docker

$ sudo pacman -S docker

Start and enable the service

$ sudo systemctl start docker.service
$ sudo systemctl enable docker.service

Hello World

After you have installed docker, run this command as a test

$ sudo docker run hello-world

If it completes successfully, you can follow along


The Network

The network will be called vulnerable. It will have a 10.0.0/24 subnet

Create it with this command

$ sudo docker network create vulnerable --attachable --subnet 10.0.0.0/24

Attacker Container

For this, I will use Parrot OS. It’s docker images are better Kali Linux Images.

First download the Parrot OS Docker image. This command will take a while depending on your internet connection.

$ docker pull parrotsec/security:latest

Create and run the container .

sudo docker run \
    --name parrot \
    -it \
    --hostname parrot \
    --network vulnerable \
    --ip="10.0.0.2" \
    --env DISPLAY=$DISPLAY \
    -v /dev/shm:/dev/shm \
    --device /dev/snd \
    --device /dev/dri \
    --mount type=bind,src=/tmp/.X11-unix,dst=/tmp/.X11-unix \
    parrotsec/security:latest \
    /bin/bash

All tools available in Parrot OS are now an apt-get away.

Use this command to restart the parrot OS container after a reboot.

$ sudo docker start -a parrot

Target container:Metasploitable2

This is a very vulnerable test machine. It is what I recommend for anyone starting out.

Open another terminal and pull the metasploitable image. The image is around 500MB.

$ docker pull tleemcjr/metasploitable2

To run a metasploitable container:

docker run \
    -it \
    --network vulnerable \
    --ip="10.0.0.3" \
    --name metasploitable \
    --hostname metasploitable2 \
    tleemcjr/metasploitable2 \
    bash

You should see a terminal prompt like this

root@metasploitable2:/#

Start the vulnerable services

root@metasploitable2:/# services.sh

You can now access metasploitable from 10.0.0.3

If you want to stop the container, close the terminal with CTRL + D

Run this command to start metasploitable again

$ sudo docker start -a parrot

Then start the vulnerable services.

root@metasploitable2:/# services.sh

Guides & Tutorials

There are tons of free guides out there on metasploitable.

1. The Easiest Metasploit Guide You’ll Ever Read
2. Metasploit Unleashed
3. Metasploitable 2 Exploitability Guide
4. Youtube Tutorials

If you don’t know what guide to use, I recommend this one.

More vulnerable containers 🧑‍💻

You can extend the lab with any of these containers depending on your learning goal.

OWASP Juiceshop

This container focusses on web application security.

To create and start the juiceshop container for the first time

docker run -d \
    --name juiceshop \
    --network vulnerable \
    --ip="10.0.0.6" \
    bkimminich/juice-shop

Check if it is running

$ docker ps

Access the web interface from this URL

http://10.0.0.6:3000/

Stop the container when you’re done

docker stop juiceshop

Start the container again

docker start juiceshop

Juiceshop Guides

• Pwning Juiceshop
• Youtube Videos

OWASP Webgoat 🐐

Webgoat is a ctf-style vulnerable container focused on web application security.

Create and run the container for the first time

docker run  -d \
    --name webgoat \
    --network vulnerable \
    --ip="10.0.0.4" \
    -e TZ=$(cat /etc/timezone) \
    webgoat/goatandwolf

Access Webgoat and Webwolf from these URLs

10.0.0.4:8080/WebGoat

10.0.0.4:9090/WebWolf

To stop the container

docker stop webgoat

To Start the container again.

docker start webgoat

If you can’t access the url, check if it is running.

$ docker ps -a

Why I use docker for a pentest lab

Two Operating systems make my computer painfully slow. Containers aren’t resource-intensive and perform well. This fits my use case.

If you have RAM to spare then it’s really not that much of a difference.

When not to use Docker

If you want to run Windows containers on a Linux host. You can run linux containers on WIndows though

Common Docker Commands

Stop a container:

$ sudo docker stop containerName

Start a container

$ sudo docker start containerName

List running and stopped containers

$ sudo docker ps -a

Spawn a bash shell in a running container

$ sudo docker exec -it containerName bash

Docker has tab completion for each of these commands.

Graphical apps inside docker

Sometimes you may want to run a GUI tool like firefox or burpsuite.

The Parrot OS commands above are already set for running graphical apps. You only need to install these packages

$ apt install hicolor-icon-theme \
    libcanberra-gtk* libgl1-mesa-dri \
    libgl1-mesa-glx libpangox-1.0-0 \
    libpulse0 libv4l-0 fonts-symbola \

You can run a few commands to avoid some errors encountered when running GUI apps

Burpsuite

Burp Suite is a web app pentesting tool for monitoring http requests and responses.

To install and run burpsuite inside the parrot os container.

# sudo apt update
# sudo apt install burpsuite
# java -jar -Xmx2G /usr/bin/burpsuite

You can then point your browser to use 10.0.0.2:8080 as the proxy and burp will intercept everything

Firefox

Firefox, is a free and open-source web browser.

To install and run firefox:

$ apt install firefox ca-certificates

Credits

Illustrations from Freepik

Further reading/research

Jess Frazelle has written an awesome blog post with details on running graphical apps inside Docker containers. She’s also given this awesome Talk/Demo on running various applications and retro games inside docker containers.

Updated version

This website costs me almost nothing to run. I take advantage of a few free services. This site is fast and looks okay. Here’s what I use.

1. DNS : Cloudflare

DNS is what enables the translation of website names to IP addresses. I use Cloudflare to manage the DNS of my domain at ianmuchina.com. I can have unlimited subdomains.

2. CDN: Cloudflare

Cloudflare have over 200 datacenters worldwide meant for just this purpose.

When using Cloudflare’s Content Delivery Network, someone in Nairobi gets the site assets from a datacenter in Mombasa instead of somewhere further like France or USA.

A content delivery network greatly improves speed.

4. Framework : Jekyll

I use Jekyll to generate the static files of my website. Jekyll has some awesome free themes. However it can take a bit of time to configure for those not familiar with it.

5. Hosting: Netlify + GitHub

Netlify is what builds my site from the configuration files of my site. I use netlify with github. The static content is regenerated anytime I modify the config files

6. Domain Registrar - Cloudflare

One gets a domain from a domain registrar. You can get a free one from Freenom. I have a freenom domain that I dont use at muchina.gq, but I abandoned it in favor of ianmuchina.com. Most .com domains are not that expensive and cost around $12 per year.

7. SSL: Cloudflare

A site with SSL correctly configured has a padlock alongside the domain name. When using the cloudflare CDN, your site automatically gets free SSL

TLDR

Take advantage of Cloudflare & Github Pages and just pay for the domain._

What is SSH?

SSH is a protocol used by system administrators and developers to communicate securely with a server. Authentication can be done with SSH keys or passwords.

Passwords and why they’re not recommended

We all know passwords and have used them at some point. Those secret string of characters you have to remember to prove your identity.

Passwords are good and better than nothing, but securing a server with just a password is not recommended especially on critical systems.

1. Weak passwords can be brute-forced
2. Sometimes passwords have to be shared for group accounts.
3. Only one compromised server is enough to phish all user credentials.
4. Revocation of compromised user passwords is not easy or convenient
5. Without a second factor, they are a single point of failure

You can improve the security by using tools like fail2ban to block IP addresses that have multiple failed login attempts.

Why use keys?

Keys are the default authentication method on most cloud providers eg: Digital Ocean. There is a good reason for this. Keys are more secure than passwords

SSH keys are a matching set of cryptographic keys which can be used for authentication.

Each set contains a public and a private key. The public key can be shared freely without concern, while the private key must be vigilantly guarded and never exposed to anyone. source

Why SSH Keys are more secure

• One user account can have multiple keys. This makes auditing easier
• Keys can be password protected. Note the password is just for unlocking the key and never sent to the server
• Keys can be combined with passwords
• Keys can be revoked

Keys are created with the ssh-keygen command.

When creating keys, it is a good idea to encrypt them with a password. So that even if the key is leaked, the password will add another layer of security.

Managing SSH keys

To connect to a server using keys, a user’s public key has to be in the authorized_keys file on every server they want to connect to.

This method of managing keys is the most common and works okay for small teams but it can be a burden for large organizations with many users.

But growth creeps upon you, and before you know it you are securing significant assets with sub-standard methods

Key management issues

• The authorized_keys file on all servers has to be maintained regularly
• Issuing a new key is a long process
• Losing a key locks you out

The task of updating the authorized_keys file can and should be automated for large companies. This method “works” but there is a much better alternative

SSH Certificates

This is where public keys are signed by a Certificate Authority(CA). This method scales very well across thousands of machines. It can be used for both user and host verification. This method eliminates the Trust on first use requirement.

Advantages of certificates

SSH certificates are underrated and awesome in my opinion. They have some good features:

• Server configuration is done only once
• Updating the authorized_keys file on every server is not needed.
• Audits are easier since every key is unique
• Works perfectly for giving users access to offline/air-gapped servers
• Keys can expire

Available Metadata

The signed key can contain the following useful information

1. Key Expiry date & time

This is how long a key is valid. The value can range from minutes to years. Netflix set their certificates to expire after two minutes. That’s enough time for a developer to login into a server and start an ssh session. Note that the session does not terminate when a certificate expires, only when a user logs out

2. Principals

Principals control which user accounts and hosts a user has access to. For example, there can be a root_staging principal which allows users to login as root on a staging server

You can Parse this data offline with ssh-keygen -Lf /Path/To/Signed_Cert

You can also use this site

Open Source Implementations

There are many solutions to giving users SSH access. All these are available on GitHub. Note that documentation is not that good for some tools though

1. Keybase Certificate Authority bot

This is a keybase chatbot that signs public keys. it is cli based. The bot doesn’t need any ports open.

• Used internaly @ Keybase
• Only supports keybase Authentication
• Good Documentation

Blog post on keybase SSH

2. Cashier

• Controlled via a web interface
• Integrates well with LDAP or Active Directory
• Google, Github or Gitlab login available
• Key revocation page

Video on The Tool

3. BLESS - By Netflix

• Used internally @ Netflix,
• Runs on AWS Lambda
• Video on the tool

4. SSHrimp - by Stoggi

• Runs on AWS Lambda
• Authenticate with OpenID connect
• Still in Beta

5. Keymaster - Symantec

• Supportts Oauth2 login
• Integrates well with LDAP/Active Directory
• Made by Symantec
• Documentation

More Tools

• Smallstep CA
• Uber’s PAM module

Video Resources

For those that prefer videos to blog posts

• Public key Cryptography Explained
• The Key Exchange Problem
• Conference Talks on SSH Certificates

Further Reading

I’m not the first one to write about this topic, so here’s related content covering it

Blog Posts

• How SSH is secured At Facebook
• SSH at scale - CAs and Principals
• How to SSH Properly
• If you’re not using SSH certificates you’re doing SSH wrong

Whitepapers and Manuals

• Redhat Deployment Guide
• NIST whitepaper on SSH automation
• IETF Memo on recomended practice
• ssh-keygen manual

Image Credits

Illustrations from Stories by Freepik

1. The browser method

This is done entirely online. Open a browser, go to a upload a song and get vocal and instrumental stems. This is a simple method which relies on third party websites to process the music.

Steps

The only requirement is a browser.

1. Go to moises.ai
2. Create an account
3. Upload a song
4. Wait for the isolation to be done
5. Enjoy !!

Advantages

• Can be done from any device with a browser even a mobile phone
• Requires no special knowledge
• Easy user interface

Disadvantages

• Only 5 uploads per month
• There’s a queue that may affect your waiting time

You can use https://ezstems.com as an alternative for more upload options and it works in a simmilar way.

2. The App Method

This second method is simmilar to the first, but only uses the computer’s resources.

Steps

This method only works on windows

1. Download and install the latest version of Python
2. During the installation make sure add to path checkbox is selected.
3. Download SpleetGUI
4. Extract the zip file
5. Install SpleetGUI

*I have no idea if it is safe to use

3. Command line Method

This last is done entirely from the command line and is great for those familiar with the command line. For this method, you will require git and conda.

After they are both installed, type the following commands to install spleeter.

$ git clone https://github.com/Deezer/spleeter
$ conda install -c conda-forge spleeter

To run spleeter:

$ python3 -m spleeter -i your_song.mp3 -p spleeter:2stems -o output

How Does it Work?

Machine learning is artificial intelligence that can automatically learn from past data. You give it enough data and it can literally learn like humans, through practice

For more detailed information, check out the spleeter documentation

Conclusion

This is a great tool that can be used to get clean vocals from almost any song. It also works well to de-noise a track.

In this article, I will go over how to create and deploy these tokens.

Requirements

• Email Address
• Target With An Internet Connection

Steps

1. Creating the file

• Go to canarytokens.org
• Choose either a Word Document or an Adobe PDF token
• Enter an email address where alerts will be sent
• Choose an appropriate reminder eg: “Opened”
• Click on “Create My Canary Token”
• Download the token

2. Setting up the file

After Downloading the file you should:

• Rename the file
• Edit the content of the file
• Send the file to someone
• Wait for an alert

3. Receiving Alerts

The alerts contain:

• The IP address the file was opened from
• A Reminder if you wrote one
• The time and date when the file was opened

If someone with internet access opens the document, you will be alerted. Usualy within seconds.

Further Reading

Official Documentation