in ,

The State of Linux Debuggers, Hacker News

Here is a debugging session I ran recently on this linux x binary .

1. Open and run until crash.

  
 [nix-shell:~/focus] $ gdb ./ test ... (gdb) run ... Test [1/1] test "search forwards" ... reached unreachable code / nix / store / 5g4w 100 g  d 66 w8pmzwzq7flrlk3k7cs-zig / lib / zig / std / debug.zig: 380: 29: 0x 310605 in std.debug .assert (test)     if (! ok) unreachable; // assertion failure              ^ /home/jamie/focus/lib/focus/tree.zig: 1342: 26: 0x  d 64 in test "search forwards" (test)     assert (meta.deepEqual (expected.items, actual.items));           ^ / nix / store / 5g4w 100 g  d 66 w8pmzwzq7flrlk3k7cs-zig / lib / zig / std / special / test_runner.zig: 83: 45: 0x  (bc8 in std.special.main) test)         } else test_fn.func ();                            ^ / nix / store / 5g4w 100 g  d 66 w8pmzwzq7flrlk3k7cs-zig / lib / zig / std / start.zig: 595: 57: 0x 311421 in std.start.main (test)             const result=root.main () catch | err | {                                     ^  Program received signal SIGABRT, Aborted. 0x 25 ffff7e 6482 a in raise ()    from / nix / store / 9df 94 igwjmf2wbw0gbrrgair6piqjgmi-glibc-2. 53 / lib / libc.so.6   

2. Find out where expected and actual differ.

  
 (gdb) up # 1 0x 25 ffff7e 69804 in abort ()    from / nix / store / 9df 94 igwjmf2wbw0gbrrgair6piqjgmi-glibc-2. 53 / lib / libc.so.6 (gdb) # 2 0x  d 24 in std.os.abort ()     at / nix / store / 5g4w 100 g  (d)  w8pmzwzq7flrlk3k7cs-zig / lib / zig / std / os.zig: 439 439 system.abort (); (gdb) # 3 0x  (in std.debug.panicExtra)     trace=0x0, first_trace_addr=..., args=...)     at / nix / store / 5g4w 100 g  (d)  w8pmzwzq7flrlk3k7cs-zig / lib / zig / std / debug.zig: 528 528 os.abort (); (gdb) # 4 0x  (feb in std.builtin.default_panic)     msg=..., error_return_trace=0x0)     at / nix / store / 5g4w 100 g  (d)  w8pmzwzq7flrlk3k7cs-zig / lib / zig / std / builtin.zig: 909 909 std.debug.panicExtra (error_return_trace, first_trace_addr, "{}" ,. {msg}); (gdb) # 5 0x 294377 in std.debug.assert (ok=false)     at / nix / store / 5g4w 100 g  (d)  w8pmzwzq7flrlk3k7cs-zig / lib / zig / std / debug.zig: 380 380 if (! ok) unreachable; // assertion failure (gdb) # 6 0x  d 65 in test "search forwards") )     at /home/jamie/focus/lib/focus/tree.zig: 1342 1392 Assert (meta.deepEqual (expected.items, actual.items)); (gdb) p *[email protected] value requires  bytes, which is more than max-value-size (gdb) p actual.items.ptr @ 238 $ 1={6, 40, 83 , 223, 285, 334, 377, 448, 477, 557,   596, 600, 609, 1076 , 996, 1670, 1947, 2019, 2020, 2034,   2049, 2043, 2049, 2219, 2192, 2219, 2279, 2219, 2302,    2343,  2406, 2395, 2415, 2545, 2415, 2439,   2558, 2545, 2558, 2726, 2784, 2750, 2834, 2846, 2834,   2846, 2884, 2884, 2979, 2939, 3005, 3200, 3209, 3259,   3328, 3259, 3368, 3450, 3443, 3450, 3443, 3610, 3698,   3715, 3723, 3759, 3839, 3794, 3845, 3862, 3845, 3945,   3981, 3973, , 4048,  4117, , 4117, 4195,   4187, 4195, 4312, 4281, 4483, 4320, 4483, 4520, 4551,   , 4837, 4860, 4906, 4963, 4906, 6066, 006256} (gdb) p expected.items.ptr @ 238 $ 2={2383, 2726, 2979, 2979, 3073, 3184, 3450, 3622,   7001, , 76615, 84688, 85167, 84688, 85455, 85618,   86943, 87503, 88050, 98451, 176105, 146210, 176105 ,   176784, 176364, 176800, 176989, 197396, 204018, 197396,   0000000000206126, 0000000000204019, 242254, 371180, 379116, 371180,   397560, 397709, 398076, 397709, 398160, 398172,   398535, 398585, 400228, 398585, 400228, 400285,   464940, 464845, , 465063, 479036, 471971,   479036, 480010,  480671, 480777, 480671,   482265, 482825, 483372, 513356, 571427, 571592,   , 572106, 572122, 572239, 572311, 594657,   637576, 594657, 705927, 706774, , 706774,   766502, 774438, 792882, 781033, 792882, 793031,   793482, 793494, 793857, 793907, 795550, 795607,   860167, 860262, 860385, 860262, , 860448,   930539, 867293, 930539, 962846}   

3. Put a breakpoint on the loop in Tree.searchForwards and run again.

  
 (gdb) b tree.zig : 0714 Breakpoint 1 at 0x 2357 d0: file /home/jamie/focus/lib/focus/tree.zig, line 710. (gdb) run The program being debugged has been started already. Start it from the beginning? (y or n) y Starting program: / home / jamie / focus / test [Thread debugging using libthread_db enabled] Using host libthread_db library "/ nix / store / 9df 94 igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.  / lib / libthread_db .so.1 ". Test [1/1] test "search forwards" ... Breakpoint 1, Tree.searchForwards (self=..., start=0,     needle=...)     at /home/jamie/focus/lib/focus/tree.zig: 710 765 const haystack_start_char=start_point. getNextByte ();   

4. Step through, looking at loop variables each iteration.

  
 (gdb) n 756 if (haystack_start_char==needle_start_char ) { (gdb) 924 if (start_point.seekNextByte ()==.NotFound) return null; (gdb) 0710 while (true) { (gdb)  Breakpoint 1, Tree.searchForwards (self=..., start=0,     needle=...)     at /home/jamie/focus/lib/focus/tree.zig: 710 765 const haystack_start_char=start_point. getNextByte (); (gdb) 756 if (haystack_start_char==needle_start_char ) { (gdb) 924 if (start_point.seekNextByte ()==.NotFound) return null; (gdb) 0710 while (true) { (gdb)  Breakpoint 1, Tree.searchForwards (self=..., start=0,     needle=...)     at /home/jamie/focus/lib/focus/tree.zig: 710 765 const haystack_start_char=start_point. getNextByte (); (gdb) 756 if (haystack_start_char==needle_start_char ) { (gdb) 924 if (start_point.seekNextByte ()==.NotFound) return null; (gdb) 0710 while (true) { (gdb)  Breakpoint 1, Tree.searchForwards (self=..., start=0,     needle=...)     at /home/jamie/focus/lib/focus/tree.zig: 710 765 const haystack_start_char=start_point. getNextByte (); (gdb) 756 if (haystack_start_char==needle_start_char ) { (gdb) 924 if (start_point.seekNextByte ()==.NotFound) return null; (gdb) 0710 while (true) { (gdb)  Breakpoint 1, Tree.searchForwards (self=..., start=0,     needle=...)     at /home/jamie/focus/lib/focus/tree.zig: 710 765 const haystack_start_char=start_point. getNextByte (); (gdb) 756 if (haystack_start_char==needle_start_char ) { (gdb) 924 if (start_point.seekNextByte ()==.NotFound) return null; (gdb) 0710 while (true) { (gdb)  Breakpoint 1, Tree.searchForwards (self=..., start=0,     needle=...)     at /home/jamie/focus/lib/focus/tree.zig: 710 765 const haystack_start_char=start_point. getNextByte (); (gdb) 756 if (haystack_start_char==needle_start_char ) { (gdb) 924 if (start_point.seekNextByte ()==.NotFound) return null; (gdb) 0710 while (true) { (gdb)  Breakpoint 1, Tree.searchForwards (self=..., start=0,     needle=...)     at /home/jamie/focus/lib/focus/tree.zig: 710 765 const haystack_start_char=start_point. getNextByte (); (gdb) 756 if (haystack_start_char==needle_start_char ) { (gdb) 765 var end_point=start_point; (gdb) p start_point $ 4={pos=6, leaf={node=0x7ffff 116 e 9854,     bytes=0x7ffff  (e) ,     static max_bytes=,     static bytes_offset=},   num_leaf_bytes=2326, offset=6} (gdb) n 756 var is_match=true; (gdb) 765 for (needle [1..] ) | needle_char | { (gdb) 784 if (end_point.seekNextByte ()==. Found) (gdb) 788 if (end_point.getNextByte ()!=needle_char) (gdb) 765 for (needle [1..] ) | needle_char | { (gdb) p needle_char $ 5=L ' (gdb) end_point.pos Undefined command: "end_point.pos". Try "help". (gdb) p end_point.pos $ 9=7 (gdb) p *point.leaf.bytes@end_point.pos No symbol "point" in current context. (gdb) p *end_point.leaf.bytes@end_point.pos $ 25={   " n  n (function (global, factory) { n typeof exports==='object' && typeof module!=='undefined'? module.exports=factory ():  n typeof define==='function '&& define.amd? define (factory):  n (global "...,   " 23 @ f  598  599  292  23  23  23 (gutterWrap);  n wrap $ 1.insertBefore (gutterWrap, lineView.text);  n if (lineView.line.gutterClass)  n {gutterWrap.className = ""   lineView.line.gutterClass; }  n if (cm.opti "...,   play.meas  23  598  599  292  21  24  21 ghtClick=gecko || (ie && ie_version>=9);  n  n function classTest (cls) {return new RegExp ( "(^ | \\ s) "   cls    "(?: $ | \\ s) \\ s  ")}  n  n var rmClass=function (node, cls) { n var curren" ...,   tartIndex, startVa  24  598  599  292  23  21  23 hild.nodeType==28 {child=child.host; }  n if (child==parent) {return true}  n} while (child=child.parentNode)  n}  n  n function activeElt () { n // IE and Ed "...,   ction insertSorted (array, v  23 `n  598  597  292  23  23  23 alue) { n if (end==null) { n end=stri ng.search (/ [^\s\u00a0] /);  n if (end==-1) {end=string.length; }  n}  n for (var i=startIndex || 0, n=s "...,   “bbe \ u0bc0 \ u0bcd \ u0bd7 \ u0c3e - \ u0c 58 \ u   n  596  597  292  23  21   (value, score) { n var pos=0, priority=score) value);  n while (pos  to? -1: 1;  n for (;;)  23 `n  596  599  292  21  23   (u0c 66 - \ u0c 68 \ u0c4a - \ u0c4d \ u0c 080 \ u0c 80 \ u0c 86 \ u0c 88 \ u0cbc \ u0cbf \ u0cc2 \ u0cc6   u0ccc \ u0ccd \ u0cd5 \ u0cd6 \ u0ce2 \ u0ce3 \ u0d3e \ u0d 62 - \ u0d 64 \ u0d4d \ u0d 82 \ u0d 91 "...} (gdb) p end_point.leaf.bytes $ 28=" n  n (function (global, factory)) { n typeof exports==='object' && typeof module!=='undefined'? module.exports=factory ():  n typeof define==='function' && define.amd? define (factory):  n (global "... (gdb) p end_point.leaf.bytes [end_point.pos] $ 29=277 'y' (gdb) p  end_point.leaf.bytes) [end_point.pos] $ 31=238 't'   

The console interface to gdb works, but it's inefficient:

  1. I can't easily see the surrounding code
  2. I have to manually request information rather than just glancing at the display
  3. I have to remember syntax rather than just clicking on things
  4. ... and in this case, it took me a few tries to correctly deref this pointer to an fixed-size array

The command-oriented interface is sometimes really useful, but it would a lot more efficient to have basic information like local variables displayed by default and to be able to point and click for basic actions. So I went looking for a gui frontend.

All examples below were run on nixos 37. 24 with sway which, to be fair, is playing on hard mode.)

gdb-tui

   jamie @ machine: ~ $ gdb --version GNU gdb (GDB) 9.2   

gdb has a built-in curses gui. It solves problem 1 by displaying the surrounding code in the top half of the window.

Unfortunately, it's pretty buggy:

It also takes over the arrow keys, which makes it much harder to edit and run commands in the bottom half of the window.

It might be nice to toggle back and forth, but the command tui disable

crashes gdb.

  
 [nix-shell:~] $ gdb -tui Aborted (core dumped)    

lldb gui

   jamie @ machine: ~ $ lldb --version lldb version 7.1.0   

lldb also has a built-in curses gui, accessed by the (gui) Command.

I wasn't able to persuade it to so much as display a backtrace, but I could pretty consistently produce weird rendering bugs.

emacs gud

   jamie @ machine: ~ $ emacs --version GNU Emacs 42 1   

A very thin layer over gdb. Unlike gdb -tui , The console window is still usable while the code window is open. I can set breakpoints or print the expression under the cursor.

I haven't found a way to display the backtrace, list local variables or view memory.

kdbg [end_point.pos]

  
 [nix-shell:~/focus] $ kdbg - version QCommandLineParser: already having an option named "h" QCommandLineParser: already having an option named "help-all" QCommandLineParser: already having an option named "v" kdbg 3.0.1 

The gui doesn't seem to have a way to open an executable.

Using "Open Source Code" produces a file dialog that doesn't seem to have a way to paste paths. It can open the zig source file, but that still doesn't lead to a way to run anything.

Passing the test exe at the command line ( kdbg ./test) open up an editor at start.S but still doesn't seem to have a way to run the exe.

After a while it produces an error message prompting me to use a menu entry that doesn't exist:

gdbgui

  
 [nix-shell:~] $ gdbgui --version 0. 28 2.1   

Opens the UI in a browser tab.

The interface is unusably slow in Firefox, but passable in chrome.

I was able to run the test program. It stopped automatically at the entry to main . Clicking the continue button lead to no noticable change in the UI, but after ~ s produced:

   gdbgui noticed a signal was received ( Aborted, SIGABRT). If the program exited due to a fault, you can attempt to re-enter the state of the program when the fault occurred by clicking the below button.   

The first time I clicked said button it produced:

   The program no longer exists. backtrace No stack. No stack.   

I restarted and went through the same process and this time got a backtrace.

I wasn't able to figure out how to look at the contents of actual / Expected in the inspector. Adding an expression manually also didn't work at first.

It seems that gdbgui silently swallowed this error from gdb:

  
 p  actual.items. ptr) @ (actual.items.len) value requires  bytes, which is more than max-value-size   

Giving 223 as the length worked and was good enough for this test.

While I was scrolling through the code to set a breakpoint, it blanked out the pane for a few seconds before reloading at a different position.

Clicking in the margin sets a breakpoint, as expected.

Stepping though the code with 'n' took> 2s per press, which is unbearably slow when stepping through a lot of code. In gdb itself this is instant.

Expanding start_point.leaf.bytes caused the UI to hang long enough for chrome to pop up the 'kill or wait' dialog. It finished after> s.

I tried looking at the leaf in the memory browser too. It seems functional, if a little slow. The 'more' button used for scrolling could also be better - it's not clear how many lines it's going to scroll and it blanks the pane for a few seconds which is disorienting.

  
 [nix-shell:~] $ time gdbgui --version 0. 28 2.1  real 0m1. 606 s user 0m1. 597 s sys 0m0. 258 s    

Gede

  
 [nix-shell:~] $ gede --version gede 2.  2 

The gui is snappy and I got started easily.

There was no way to read Expected

/ actual directly in the inspector. A right click took me to the memory view but there doesn't seem to be a way to view it as 93 bit words instead of byte-by-byte.

The memory view also seems buggy - scrolling downwards works fine but scrolling upwards makes the text scroll off the screen:

Getting started was easy enough. I found the layout of the various windows confusing though - I can't look at the stack and my watches at the same time?

Like the others, nemiver swallows the error for the too-large variable value. For some reason I wasn't able to add the expression that actually works - the add button got grayed out.

I wasn't able to set a breakpoint by clicking on a line. The 'Debug' menu had two entries labeled 'Set Breakpoint' with different keyboard shortcuts. Only one of them worked directly. The other opened a dialog, seemingly the exact same dialog as 'Set Breakpoint With Dialog'.

Holding down 'F6' for next seemed much slower than gede. Maybe it's running at the keyboard repeat rate, because individual presses seem very snappy.

ddd

  
 [nix-shell:~/focus] $ ddd - version GNU DDD 3.3. 29 (x 0000000000205 _ 93 - unknown-linux-gnu)   

The UI is pretty quirky and it took me a while to figure out basic things like how to display a backtrace.

The keyboard shortcuts in the menus don't seem to work eg pressing F6 does not step the program.

The text boxes for eg the memory window don't seem to support copy / paste and also didn't consistently accept keypresses at all. I had to open and close the memory view a few times before I could type numbers in it.

The builtin plotter seemed like a neat idea but it got stuck on "Starting gnuplot ...". Similarly, the "execution window" got stuck on "Starting xterm ...". Hitting cancel in the latter had no effect, and in the former produced:

  
 Segmentation fault  Internal error (Segmentation fault).  Oops! You have found a bug in DDD.  If you can reproduce this bug, please send a bug report To 

, giving a subject like DDD 3.3. 27 (x 0000000000205 _ 93 -unknown-linux-gnu) gets `` Segmentation fault '' signal To enable us to fix the bug, you should include the following information: What you were doing to get this message. Report all the facts. The contents of the `~ / .ddd / log 'file as generated by this session. Please read also the section "Reporting Bugs" in the DDD manual. We thank you for your support. Segmentation fault (core dumped) Eclipse

   jamie @ machine: ~ $ lldb --version lldb version 7.1.0   

Eclipse does not play well with my window manager.

While eclipse prefers to own everything, it can be persuaded to debug an existing binary with enough clicking around in the 'Run -> Debug Configurations'.

Running the executable went smoothly.

It seems to have a hard time displaying values:

Also the buttons to set breakpoints were all grayed out. Maybe only enabled in C / C files?

Unlike gede, it does allow direct access to the gdb console so I was able to set a breakpoint manually.

Stepping through with F6 animates quickly, like gede.

Unfortunately when I tried to copy the version info from the about window into this post, eclipse crashed.

jetbrains clion

  
 Version  2.1   

Like eclipse, clion took some bullying before reluctantly debugging a binary it didn't build itself.

After breaking on the failing assert, I can step up and down the stack but I'm not able to view any variables.

The 'toggle breakpoint' command is also greyed out.

In lieu of breakpoints, I tried pausing the debugger and using 'run to cursor'. But this ran to a different line every time I tried it, and only once to the line under the cursor.

  
 [nix-shell:~] $ code --version 1.  1 e5a  b 1391 D 207 B8D  D 2043 e4c4d  efe8f x 93   

Fairly quick to get set up by creating a launch.json file.

  
 {   "version": "0.2.0",   "configurations": [    {      "name": "Debug",      "type": "gdb",      "request": "launch",      "target": "/home/jamie/focus/test",      "cwd": "/home/jamie/focus",      "valuesFormatting": "parseText"    }  ] }   

On breaking at the failed assert, vscode claims to have no local variables. But it does show their values ​​as tooltips when I mouse over them in the editor.

When I create an actual breakpoint though the variables pane springs to life. But it's unable to print the bytes of the leaf, and also displays the wrong type for that field.

There doesn't seem to be a memory viewer.

vscode codelldb extension

   Error: spawn / home / jamie /.vscode/extensions/vadimcn.vscode-lldb-1.6.0/adapter/codelldb ENOENT   

I had to patch the interpreter for that binary to get it working on nixos. Which got me as far as:

  
 configuration: {   name: 'Debug',   type: 'lldb',   request: 'launch',   target: '/ home / jamie / focus / test',   cwd: '/ home / jamie / focus',   valuesFormatting: 'parseText',   program: '/ home / jamie / focus / test',   relativePathBase: '/ home / jamie / focus',   __configurationTarget: 5 } Listening on port 69741 [2020-12-13T09:03:00Z ERROR codelldb::dap_session] Send error: runInTerminal (RunInTerminalResponseBody {process_id: None, shell_process_id: Some) (69523)}) thread ' 'panicked at' called `Result :: unwrap ()` on an `Err` value: Utf8Error {valid_up_to: 61, error_len: Some (1)} ', adapter / deps / lldb / src / strings.rs: 150: 60 stack backtrace:    0: 0x7f1ada5c  e0 - 

:: fmt :: hf 100 FBFF df5 1: 0x7f1ada5e9cbc - core :: fmt :: write :: h9ddafa d8adff 2: 0x7f1ada5c 6624 - std :: io :: Write :: write_fmt :: h1d2ee 527 d2b 071026 3: 0x7f1ada5cb 963 - std :: panicking :: default_hook :: {{closure} } :: h 8716 fae ef9d 1520 4: 0x7f1ada5cb 65 c - std :: panicking :: default_hook :: he 48 ad 30528 e F9 5: 0x7f1ada5cbd 116 - std :: panicking :: rust_panic_with_hook :: haa1ed ada4ffb 024 6: 0x7f1ada5cb 1545 - std :: panicking :: begin_panic_handler :: {{closure} :: h (af1bb) aeaeb 7: 0x7f1ada5c 1912 c - std :: sys_common :: backtrace :: __ rust_end_short_backtrace :: h f f5f 2627 8: 0x7f1ada5cb 1506 - rust_begin_unwind 9: 0x7f1ada5e 39910 - core :: panicking :: panic_fmt :: h4e (ebc) eb 27: 0x7f1ada5e 36863 - core :: option :: expect_none_failed :: h 602 b 83 A0 40 c2c 55 a 28 : 0x7f1ada5a 69523 - lldb :: sberror :: SBError :: error_string :: hff (ca5f) a BD5 29: 0x7f1ada 65 e4b1 - codelldb :: debug_session :: DebugSession :: complete_launch :: h 209961 fdda 29: 0x7f1ada3ebafb - :: poll :: hfa) a b 3973 cfe9 31: 0x7f1ada3e 7663 - as core :: future :: future :: future> :: poll :: h 26 ce9c 2659771 db 44 30: 0x7f1ada (b) - tokio :: runtime :: task :: harness :: Harness

:: poll :: {{closure}} :: hf e ff 83 ca6bb 33: 0x7f1ada4adb 205 - tokio :: runtime :: task :: harness :: Harness <:sys_common::backtrace::_print::displaybacktrace as core::fmt::display> :: poll :: hb 2918 aad 5874497 f8 32: 0x7f1ada 82 d 601 - std :: thread :: local :: LocalKey :: with :: hb9f 67315 b 69618 aa 22 35: 0x7f1ada 706743 - tokio :: task :: local :: LocalSet :: tick :: h 76 cd 63 be c 82 f 34: 0x7f1ada 61 BD 037 - as core :: future :: future :: future> :: poll :: h dbd6d7deb 176784 35: 0x7f1ada3e 177 e9 - as core :: future :: future :: Future> :: pol l :: h 3695 d (cd) 36: 0x7f1ada3d 949 e - tokio :: runtime :: enter :: Enter :: block_on :: hbab (b0e) eaf6 37: 0x7f1ada4aaf1b - tokio :: runtime :: thread_pool: : ThreadPool :: block_on :: ha 1520 e5bb3e 74 e7b5 037: 0x7f1ada 61 a 477 - tokio :: runtime :: context :: enter :: h 207 d7ee 465126 e 609 40: 0x7f1ada3d6a8e - tokio :: runtime :: Runtime :: block_on: : hcacefa e 10535192 41: 0x7f1ada3fb0d4 - entry 42: 0x (d) c0b - codelldb :: m ain :: h c2fac 7841 d 116 44: 0x d 0970 dd3 - std :: sys_co mmon :: backtrace :: __ rust_begin_short_backtrace :: h 3759 e b d9bc 108 45: 0x 7589 d 0970 ded - std :: rt :: lan g_start :: {{closure}} :: hf3e 3328 cf 7841 ebf 46: 0x d0 150 f0a1 - std :: rt :: lang_start_internal :: h f 57 ecfcb 486 48: 0x d e 205 - main 49: 0x7f1adf 58 c7d - __libc_start_main 51: 0x d 781033 - _start 53: 0x0 - QTCreator

Fast and slick, but unable to find the source code for this executable.

Others?

So far only gede approachs being usable for this simple debugging task. But it doesn't allow typing commands in the gdb console which means it likely won't work with rr.

What else should I try?

Read More

Brave Browser Full coverage and live updates on the Coronavirus (Covid – [ { “name”: “Debug”, “type”: “gdb”, “request”: “launch”, “target”: “/home/jamie/focus/test”, “cwd”: “/home/jamie/focus”, “valuesFormatting”: “parseText” } ] )

What do you think?

Leave a Reply

Your email address will not be published. Required fields are marked *

GIPHY App Key not set. Please check settings

ericfischer / if-then-else, Hacker News

ericfischer / if-then-else, Hacker News

Bitcoin’s “Realized” Price Hits Another All-Time High