Sunday, June 18, 2017

Manually Removing GoogleSoftwareUpdate in macOS

Wrapped it in a basic check for it's existence to avoid having to sudo if not necessary:

if [ -d "/Library/Google/GoogleSoftwareUpdate/" ]; then
  echo "sudo'ing to remove GoogleSoftwareUpdate. Might need to enter your password."
  sudo rm -rf /Library/Google/GoogleSoftwareUpdate/
  sudo rm -rf /Users/**/Library/Google/GoogleSoftwareUpdate
  sudo rm -f /Users/**/Library/Logs/GoogleSoftwareUpdateAgent.log
  sudo rm -f /Users/**/Library/LaunchAgents/com.google.keystone.agent.plist
fi

Reboot to avoid a bunch of errors in your system log. Have to keep the software up to date on your own after removing it. Do it at your own risk.

Monday, May 22, 2017

Crystal GC and "Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS Program received and didn't handle signal IOT (6)"

Using Crystal 0.22.0, sometimes the GC seems like it is slowing things down a bit or getting hung up, but it turns out that collecting more frequently doesn't seem to help much, and slows it down more overall.

When you eventually run out of memory, you'll hit (or at least I did in macOS):

Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS
Program received and didn't handle signal IOT (6)

The impact of those hangs is not remedied by doing forced cleanup, e.g.:

if GC.stats.free_bytes < 90000
  GC.collect
end

I tried that in various ways, but you can't squeeze more memory out or trick GC into running more to run faster, at least not in my testing.

I noticed in this StackOverflow answer, Mono: Too many heap sections Increase MAXHINCR or MAX_HEAP_SECTS when app takes more than 4 GB in memory, that Mono also has a problem with the Boehm GC. Perhaps the two are similar and there's a fix for Crystal or it's compilation that would help with this.

Friday, May 19, 2017

CRLF will be replaced by LF in db/structure.sql.

$ git add db/structure.sql
warning: CRLF will be replaced by LF in db/structure.sql.
The file will have its original line endings in your working directory.

Thought we had all our line-ending nonsense straightened out, but could see that structure.sql before being updated by rake db:migrate just had \n and after had some \r\n:

$ file structure.sql.before_db_migrate
before.txt: ASCII text, with very long lines
$ file structure.sql.after_db_migrate 
after.txt: ASCII text, with very long lines, with CRLF, LF line terminators
$ cat -v structure.sql.after_db_migrate
...
CREATE FUNCTION some_function_name() RETURNS void
    LANGUAGE ...
    AS $$^M
BEGIN^M
  REFRESH ...;^M
  RETURN;^M
END; $$;

So, it was in a single function of the schema. Not Rails's/ActiveRecord's fault and not a git config problem.

Friday, May 12, 2017

List Terminal Commands and Utilities and Listing All Text Editors in Posix and Linux

This is a neat command I've not used before in bash:

compgen -c | sort | uniq

That lists all of the commands available.

Getting an (incomplete) list of available text editors is a bit tougher, but we can build on the command above and guess:

compgen -c | sort | uniq | grep '^dit$\|^emacs$\|^uemacs$\|^vi$\|^vim$\|^mu$\|^nano$\|^rnano$\|^pico$\|^jed$\|^joe$'

Thursday, May 4, 2017

Working Around ActiveRecord PostgreSQL Savepoint Problems (RELEASE SAVEPOINT active_record_1)

Ended up not using it, so putting it here if someone runs into problems with AR 4.x savepoints.

The error:

ActiveRecord::StatementInvalid:         ActiveRecord::StatementInvalid: PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
        : RELEASE SAVEPOINT active_record_1
            .../gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:155:in `async_exec'
            .../gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:155:in `block in execute'
            .../gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'
            .../gems/activesupport-4.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
            .../gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
            .../gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:154:in `execute'
            ...

The fix:

module SavepointsFix
  class << self
    attr_accessor :enabled, :debug, :active_savepoints, :rollback_on_fail

    def get_backtrace
      begin
        raise
      rescue => e
        e.backtrace.join("\n")
      end
    end

  end

  def create_savepoint(name = current_savepoint_name)
    if SavepointsFix.enabled
      begin
        puts "SAVEPOINT #{name}. call stack:\n#{SavepointsFix.get_backtrace}\n\n" if SavepointsFix.debug
        result = super(name)
        SavepointsFix.active_savepoints.push name
        result
      rescue Exception => e
        puts "#{e.message}\n#{e.backtrace.join("\n  ")}" if SavepointsFix.debug
        if SavepointsFix.rollback_on_fail
          begin
            ActiveRecord::Base.connection.execute 'ROLLBACK'
            active_savepoints = []
          rescue => e
            puts "#{e.message}\n#{e.backtrace.join("\n  ")}" if SavepointsFix.debug
          end
        end
      else
      end
    else
      super(name)
    end
  end

  def exec_rollback_to_savepoint(name = current_savepoint_name)
    if SavepointsFix.enabled
      if SavepointsFix.active_savepoints.length > 0 && SavepointsFix.active_savepoints.last == name
        puts "ROLLBACK TO SAVEPOINT #{name}. call stack:\n#{SavepointsFix.get_backtrace}\n\n" if SavepointsFix.debug
        begin
          result = super(name)
          SavepointsFix.active_savepoints.pop
          result
        rescue Exception => e
          puts "#{e.message}\n#{e.backtrace.join("\n  ")}" if SavepointsFix.debug
          if SavepointsFix.rollback_on_fail
            begin
              ActiveRecord::Base.connection.execute 'ROLLBACK'
              active_savepoints = []
            rescue => e
              puts "#{e.message}\n#{e.backtrace.join("\n  ")}" if SavepointsFix.debug
            end
          end
        end
      else
        puts "IGNORED EXTRA: ROLLBACK TO SAVEPOINT #{name}. call stack:\n#{SavepointsFix.get_backtrace}\n\n" if SavepointsFix.debug
      end
    else
      super(name)
    end
  end

  def release_savepoint(name = current_savepoint_name)
    if SavepointsFix.enabled
      if SavepointsFix.active_savepoints.length > 0 && SavepointsFix.active_savepoints.last == name
        puts "RELEASE SAVEPOINT #{name}. call stack:\n#{SavepointsFix.get_backtrace}\n\n" if SavepointsFix.debug
        begin
          result = super(name)
          SavepointsFix.active_savepoints.pop
          result
        rescue Exception => e
          puts "#{e.message}\n#{e.backtrace.join("\n  ")}" if SavepointsFix.debug
          if SavepointsFix.rollback_on_fail
            begin
              ActiveRecord::Base.connection.execute 'ROLLBACK'
              active_savepoints = []
            rescue => e
              puts "#{e.message}\n#{e.backtrace.join("\n  ")}" if SavepointsFix.debug
            end
          end
        end
      else
        puts "IGNORED EXTRA: RELEASE SAVEPOINT #{name}. call stack:\n#{SavepointsFix.get_backtrace}\n\n" if SavepointsFix.debug
      end
    else
      super(name)
    end
  end
end

SavepointsFix.active_savepoints = []
SavepointsFix.debug = false
SavepointsFix.rollback_on_fail = true

::ActiveRecord::ConnectionAdapters::PostgreSQLAdapter.prepend SavepointsFix

class Minitest::Test
  alias_method :run_with_savepoint_behavior, :run

  def run
    SavepointsFix.debug = false
    result = run_with_savepoint_behavior    
    result
  ensure
    SavepointsFix.debug = false
  end

  def debug_savepoints(&blk)
    SavepointsFix.debug = true
    yield
  ensure
    SavepointsFix.debug = false
  end

  def with_savepoints_fix(&blk)
    SavepointsFix.enabled = true
    yield
  ensure
    SavepointsFix.enabled = false
  end
end

Usage in a minitest test:

with_savepoints_fix do
  some_model.destroy!
end

Thursday, March 9, 2017

Require All Ruby Files Under a File in Vanilla Ruby Without Dependencies

If you want to require all Ruby files under a particular file in a vanilla Ruby project, you can do something like:


  def self.auto_require_all
    @@pathnames_required_successfully ||= {}
    @@require_attempts ||= 0
    requires_failed = {}
    Dir["#{File.dirname(__FILE__)}/**/*.rb"].each do |pathname|
      begin
        if !@@pathnames_required_successfully[pathname] && pathname.end_with?('.rb')
          pathname_to_require = pathname.chomp('.rb')
          require(pathname_to_require)
          @@pathnames_required_successfully[pathname] = ''
        end
      rescue NameError => e
        # optionally uncomment this to show which require failed immediately
        #puts "request #{pathname.inspect} failed: #{e.message}\n#{e.backtrace.join("\n")}"
        requires_failed["request #{pathname.inspect} failed: #{e.message}\n#{e.backtrace.join("\n")}"] = ''
        @@require_attempts += 1
      end
    end

    if requires_failed.length > 0
      if @@require_attempts > 1000
        raise requires_failed.keys.join("\n")
      else
        require_tools
      end
    end
  end
  self.auto_require_all

There are gems out there to do this, but it's pretty simple to implement and you can reduce dependencies on other gems.

Friday, March 3, 2017

Passing Rest of Arguments in Bash While Retaining Quoted Arguments

If you need to pass arguments sent to a bash script to another bash script or executable, you can't just use $@ inside the string you pass in a string (e.g. to bash -c), because it will lose all the quotes around the args and will include the original. And you might want to splice the first argument off of the array without altering it. Here are some examples of how this can be done using printf to quote-escape:

send_all_args.sh

#!/bin/bash
if [ "$#" -lt 1 ]; then
 quoted_args=""
else
 quoted_args="$(printf " %q" "${@}")"
fi

bash -c "$( dirname "${BASH_SOURCE[0]}" )/receiver.sh${quoted_args}"

send_fewer_args.sh

#!/bin/bash
if [ "$#" -lt 2 ]; then
 quoted_last_args=""
else
 quoted_last_args="$(printf " %q" "${@:2}")"
fi

bash -c "$( dirname "${BASH_SOURCE[0]}" )/receiver.sh${quoted_last_args}"

receiver.sh

#!/bin/bash
for arg in "$@"; do
  echo "$arg"
done

Example usage:

$ ./send_all_args.sh
$ ./send_all_args.sh a b
a
b
$ ./send_all_args.sh "a' b" 'c "e '
a' b
c "e 
$ ./send_fewer_args.sh
$ ./send_fewer_args.sh a
$ ./send_fewer_args.sh a b
b
$ ./send_fewer_args.sh "a' b" 'c "e '
c "e 
$ ./send_fewer_args.sh "a' b" 'c "e ' 'f " g'
c "e 
f " g

(Also posted on StackOverflow.)