askiiart/engl-2311-blog
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

stuff got updated; benchmarking dwarfs is started

Browse source
This commit is contained in:
askiiart 2024-10-25 00:13:36 -05:00
parent b2895cb8ef
commit a019e5eba9
Signed by untrusted user who does not match committer: askiiart
GPG key ID: EA85979611654C30
8 changed files with 149 additions and 13 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

88
blog/benchmarking-dwarfs.html Normal file
View file

@ -0,0 +1,88 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta content="width=device-width, initial-scale=1" charset="utf-8" />
<title>Benchmarking and comparing DwarFS</title>
<link href="/style.css" type="text/css" rel="stylesheet" />
<link href="/prism.css" type="text/css" rel="stylesheet" />
</head>
<body class="line-numbers">
<h1 id="benchmarking-and-comparing-dwarfs">Benchmarking and
comparing DwarFS</h1>
<p>DwarFS is a filesystem developed by the user mhx on GitHub
[1], which is self-described as "A fast high compression
read-only file system for Linux, Windows, and macOS." One of my
ideas for blendOS was to layer different packages, and combined
with its compression and option to be mounted as a FUSE-based
filesystem, it's an appealing option for this use case - blendOS
is immutable, so it might as well have some compression.</p>
<h2 id="methodology">Methodology</h2>
<p>The datasets being used for this test will be the
following:</p>
<ul>
<li>25 GB of null data (just <code>000000000000</code> in
binary)</li>
<li>25 GB of random data<a href="#fn1" class="footnote-ref"
id="fnref1" role="doc-noteref"><sup>1</sup></a></li>
<li>Data for a 100 million-sided regular polygon; ~29 GB<a
href="#fn2" class="footnote-ref" id="fnref2"
role="doc-noteref"><sup>2</sup></a></li>
<li>The current Linux longterm release source (<a
href="https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.6.58.tar.xz">6.6.58</a>
[2]); ~1.5 GB</li>
<li>For some rough latency testing:
<ul>
<li>1000 4 kilobyte files filled with null data (again, just
<code>0000000</code> in binary)</li>
<li>1000 4 kilobyte files filled with random data</li>
</ul></li>
</ul>
<p>All this data should cover both latency and read speed
testing for data that compresses differently - extremely
compressible files with null data, decently compressible files,
and random data which can't be compressed well.</p>
<h2 id="sources">Sources</h2>
<ol type="1">
<li><a href="https://github.com/mhx/dwarfs"
class="uri">https://github.com/mhx/dwarfs</a></li>
<li><a href="https://www.kernel.org/"
class="uri">https://www.kernel.org/</a></li>
</ol>
<h2 id="footnotes">Footnotes</h2>
<section id="footnotes"
class="footnotes footnotes-end-of-document" role="doc-endnotes">
<hr />
<ol>
<li id="fn1">This data is from a very early version of a math
demonstration program made by a friend. The example below shows
what the data looks like for a 3-sided regular polygon.
<details open>
<summary>
3-sided regular polygon data
</summary>
<br>
<!-- I put it in here just as a `style`, it didn't work. I put it in as a div with that `style`, it didn't work. I put it in as a div of that class which has those properties in style.css, it works -->
<!-- i hate webdev i hate webdev i hate webdev i hate webdev i hate webdev i hate webdev -->
<div class="force-word-wrap">
<pre><code>[Vertex { position: Pos([0.5, 0.0, 0.0]), color: Col([0.5310667, 0.7112941, 0.7138775]) }, Vertex { position: Pos([-0.25000003, 0.4330127, 0.0]), color: Col([0.7492257, 0.3142163, 0.49905664]) }, Vertex { position: Pos([0.0, 0.0, 0.0]), color: Col([0.2046682, 0.25598457, 0.72071356]) }, Vertex { position: Pos([-0.25000003, 0.4330127, 0.0]), color: Col([0.6389981, 0.5204368, 0.077735074]) }, Vertex { position: Pos([-0.24999996, -0.43301272, 0.0]), color: Col([0.8869035, 0.30709425, 0.8658899]) }, Vertex { position: Pos([0.0, 0.0, 0.0]), color: Col([0.2046682, 0.25598457, 0.72071356]) }, Vertex { position: Pos([-0.24999996, -0.43301272, 0.0]), color: Col([0.6236294, 0.03584433, 0.7590722]) }, Vertex { position: Pos([0.5, 8.742278e-8, 0.0]), color: Col([0.6105084, 0.3593351, 0.85544324]) }, Vertex { position: Pos([0.0, 0.0, 0.0]), color: Col([0.2046682, 0.25598457, 0.72071356]) }]</code></pre>
</div>
</details>
<a href="#fnref1" class="footnote-back"
role="doc-backlink">↩︎</a></li>
<li id="fn2"><p>My code can generate up to 25 GB/s. However, it
does random writes to my drive, which is <em>much</em> slower.
So on one hand, you could say my code is so amazingly fast that
current day technologies simply can't keep up. Or you could say
that I have no idea how to code for real world scenarios.<a
href="#fnref2" class="footnote-back"
role="doc-backlink">↩︎</a></p></li>
</ol>
</section>
<iframe src="https://john.citrons.xyz/embed?ref=askiiart.net" style="margin-left:auto;display:block;margin-right:auto;max-width:732px;width:100%;height:94px;border:none;"></iframe>
<script src="/prism.js"></script>
</body>
<footer>
<p><a href="https://git.askiiart.net/askiiart/engl-2311-blog">Source code</a>&ensp;|&ensp;<a href="/feed.xml">RSS</a>&ensp;|&ensp;<a href="/glossary.html">Glossary</a>&ensp;|&ensp;<a href="/about.html">About</a></p>
<small>Image captions are the same as the alt text; assuming you're sighted, you can most likely ignore them.</small>
</footer>
</html>

38
blog/benchmarking-dwarfs.md Normal file
View file

@ -0,0 +1,38 @@
# Benchmarking and comparing DwarFS
DwarFS is a filesystem developed by the user mhx on GitHub [1], which is self-described as "A fast high compression read-only file system for Linux, Windows, and macOS." One of my ideas for blendOS was to layer different packages, and combined with its compression and option to be mounted as a FUSE-based filesystem, it's an appealing option for this use case - blendOS is immutable, so it might as well have some compression.
## Methodology
The datasets being used for this test will be the following:
- 25 GB of null data (just `000000000000` in binary)
- 25 GB of random data[^1]
- Data for a 100 million-sided regular polygon; ~29 GB[^2]
- The current Linux longterm release source ([6.6.58](https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.6.58.tar.xz) [2]); ~1.5 GB
- For some rough latency testing:
- 1000 4 kilobyte files filled with null data (again, just `0000000` in binary)
- 1000 4 kilobyte files filled with random data
All this data should cover both latency and read speed testing for data that compresses differently - extremely compressible files with null data, decently compressible files, and random data which can't be compressed well.
## Sources
1. <https://github.com/mhx/dwarfs>
2. <https://www.kernel.org/>
## Footnotes
[^1]: This data is from a very early version of a math demonstration program made by a friend. The example below shows what the data looks like for a 3-sided regular polygon.
<details open>
<summary>3-sided regular polygon data</summary>
<br>
<!-- I put it in here just as a `style`, it didn't work. I put it in as a div with that `style`, it didn't work. I put it in as a div of that class which has those properties in style.css, it works -->
<!-- i hate webdev i hate webdev i hate webdev i hate webdev i hate webdev i hate webdev -->
<div class="force-word-wrap">
```
[Vertex { position: Pos([0.5, 0.0, 0.0]), color: Col([0.5310667, 0.7112941, 0.7138775]) }, Vertex { position: Pos([-0.25000003, 0.4330127, 0.0]), color: Col([0.7492257, 0.3142163, 0.49905664]) }, Vertex { position: Pos([0.0, 0.0, 0.0]), color: Col([0.2046682, 0.25598457, 0.72071356]) }, Vertex { position: Pos([-0.25000003, 0.4330127, 0.0]), color: Col([0.6389981, 0.5204368, 0.077735074]) }, Vertex { position: Pos([-0.24999996, -0.43301272, 0.0]), color: Col([0.8869035, 0.30709425, 0.8658899]) }, Vertex { position: Pos([0.0, 0.0, 0.0]), color: Col([0.2046682, 0.25598457, 0.72071356]) }, Vertex { position: Pos([-0.24999996, -0.43301272, 0.0]), color: Col([0.6236294, 0.03584433, 0.7590722]) }, Vertex { position: Pos([0.5, 8.742278e-8, 0.0]), color: Col([0.6105084, 0.3593351, 0.85544324]) }, Vertex { position: Pos([0.0, 0.0, 0.0]), color: Col([0.2046682, 0.25598457, 0.72071356]) }]
```
</div>
</details>
[^2]: My code can generate up to 25 GB/s. However, it does random writes to my drive, which is *much* slower. So on one hand, you could say my code is so amazingly fast that current day technologies simply can't keep up. Or you could say that I have no idea how to code for real world scenarios.

4
blog/blendos.html
View file

@ -166,7 +166,7 @@
play in the next post.</p>
<h2 id="footnotes">Footnotes</h2>
<!-- Note if you're viewing the Markdown version: [^1] is for footnotes; that's the formatting used by Pandoc, but VS Code, for example, doesn't support it. -->
<aside id="footnotes"
<section id="footnotes"
class="footnotes footnotes-end-of-document" role="doc-endnotes">
<hr />
<ol>
@ -183,7 +183,7 @@
href="#fnref2" class="footnote-back"
role="doc-backlink">↩︎</a></p></li>
</ol>
</aside>
</section>
<iframe src="https://john.citrons.xyz/embed?ref=askiiart.net" style="margin-left:auto;display:block;margin-right:auto;max-width:732px;width:100%;height:94px;border:none;"></iframe>
<script src="/prism.js"></script>
</body>

8
blog/building-blendos.html
View file

@ -52,7 +52,7 @@
<code>archiso</code>.</p>
<p>I just added the missing packages needed to build the
<code>iso</code> file, plus all the others on blendOS's GitLab,
and rebuilt it. After a bit of troublshooting (nothing
and rebuilt it. After a bit of troubleshooting (nothing
interesting, just incompetency, wrong perms, stuff like that), I
got it working perfectly.</p>
<p>From the buid log:</p>
@ -113,9 +113,9 @@
href="https://git.blendos.co/blendOS">git.blendos.co/blendOS</a>.
Accessed 2 Oct. 2024.<br />
 Saraswat, Rudra, and Asterisk. “blendOS / image-builder ·
GitLab.” <em>GitLab</em>, git.blendos.co/blendOS/image-builder.
Accessed 3 Oct. 2024.<br />
</p>
GitLab.” <em>GitLab</em>, <a
href="https://git.blendos.co/blendOS/image-builder">git.blendos.co/blendOS/image-builder</a>.
Accessed 3 Oct. 2024sku.</p>
<iframe src="https://john.citrons.xyz/embed?ref=askiiart.net" style="margin-left:auto;display:block;margin-right:auto;max-width:732px;width:100%;height:94px;border:none;"></iframe>
<script src="/prism.js"></script>
</body>

2
blog/building-blendos.md
View file

@ -24,7 +24,7 @@ I then realized that my `iso` file worked, but was not using my repo, and I had
Turns out I had added my repo to the container's `pacman` configuration, but not to the `pacman` config for `archiso`, the tool used to generate the `iso` file, or the actual resulting `iso` image's `pacman` config either. So while I would have been able to install anything from my repo in the container, the repo was unused by `archiso`.
I just added the missing packages needed to build the `iso` file, plus all the others on blendOS's GitLab, and rebuilt it. After a bit of troublshooting (nothing interesting, just incompetency, wrong perms, stuff like that), I got it working perfectly.
I just added the missing packages needed to build the `iso` file, plus all the others on blendOS's GitLab, and rebuilt it. After a bit of troubleshooting (nothing interesting, just incompetency, wrong perms, stuff like that), I got it working perfectly.
From the buid log:

9
glossary.html
View file

@ -31,7 +31,7 @@
<li>Filesystem: The system which keeps track of how data is
written to disk, like NTFS, FAT32, or ext4. Some filesystems,
like ZFS or btrfs, have extra features like redundancy or
compression.<img src="image.png" title="alt text" alt="alt text" /></li>
compression.</li>
<li>Git: The most common version control system by far - keeps
track of different versions of files, can be used to resolve
conflicting changes, etc.
@ -91,6 +91,13 @@
<li><code>iso</code> file: A disk image file, can be "burned" to
a USB flash drive (or any other disk) and booted off of, often
used for Linux installers and/or live images.</li>
<li>FUSE: A filesystem interface used for running filesystems in
userspace (i.e. not in the kernel)</li>
<li>Kernel: The very core of an operating system, with all its
most essential functions, like filesystems (aside from those
using FUSE). On Linux, device drivers are usually in the kernel
directly, rather than being installed separately like on
Windows.</li>
</ul>
<iframe src="https://john.citrons.xyz/embed?ref=askiiart.net" style="margin-left:auto;display:block;margin-right:auto;max-width:732px;width:100%;height:94px;border:none;"></iframe>
<script src="/prism.js"></script>

4
glossary.md
View file

@ -7,7 +7,7 @@
- Container image: The base filesystem of a container
- OCI image: The OCI's standard for container images, used by essentially all Linux container platforms
- AUR: Arch User Repository, a repository for Arch Linux packages which are maintained by users. AUR only hosts computer-readable instructions and related files (via `PKGBUILD` files) for creating the packages, not the packages or programs themselves
- Filesystem: The system which keeps track of how data is written to disk, like NTFS, FAT32, or ext4. Some filesystems, like ZFS or btrfs, have extra features like redundancy or compression.![alt text](image.png)
- Filesystem: The system which keeps track of how data is written to disk, like NTFS, FAT32, or ext4. Some filesystems, like ZFS or btrfs, have extra features like redundancy or compression.
- Git: The most common version control system by far - keeps track of different versions of files, can be used to resolve conflicting changes, etc.
- Forking: Copying a Git repository and optionally and making changes to it. Can be simply to contribute the changes back to the upstream project later, or to use something as a base for your own project.
- Branches: Different "chains" of versions of code, which can, well, branch apart, merge back together, and so on. Different branches don't necessarily have to be related whatsoever, but they usually are made from other branches.
@ -29,3 +29,5 @@
- `pacman`: Arch Linux's package manager, also used by blendOS
- `PKGBUILD`: A file defining how to build a package which can be install by `pacman`.
- `iso` file: A disk image file, can be "burned" to a USB flash drive (or any other disk) and booted off of, often used for Linux installers and/or live images.
- FUSE: A filesystem interface used for running filesystems in userspace (i.e. not in the kernel)
- Kernel: The very core of an operating system, with all its most essential functions, like filesystems (aside from those using FUSE). On Linux, device drivers are usually in the kernel directly, rather than being installed separately like on Windows.

9
style.css
View file

@ -79,10 +79,11 @@ code {
padding: .125em;
}
wrap {
word-wrap: break-word
}
.force-word-wrap pre code {
white-space: normal;
word-wrap: break-word;
}
@media (max-device-width: 1200px) {
h1 {
line-height: 1.2;