Difference between revisions of "Linux utils"

From Wasya Wiki
Jump to: navigation, search
 
(91 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
= Utils =
 +
* [[ css ]]
  
 +
* [[ Git cheatsheet ]]
 +
* [[ IO Utils ]]
 +
* [[ json ]]
 +
* [[ kubernetes ]]
 +
* [[ Memory Utils ]]
  
−
* [[Pi Roles]]
+
* [[ Network Utils ]]
 +
* [[ Process Utils ]]
 +
* [[ xml ]]
  
−
==== Utils ====
+
=== Replace all occurrences of a string in a folder ===
 +
ubuntu?
 +
find /home/user/ -type f | xargs sed -i  's/a.example.com/b.example.com/g'
 +
mac:
 +
<pre>
 +
LC_ALL=C find . -type f -name "*.html" -exec sed -i '' "s/UA-47088821-3/UA-47088821-4/" {} +
 +
</pre>
  
−
* Set date
+
=== last 3 accessed files ===
−
sudo date --set "25 Sep 2013 15:00:00"
+
  find . -type f -exec stat -c '%X %n' {} \; | sort -nr | awk 'NR==1,NR==3 {print $2}'
  
−
I am able to spin up a fresh ccow system from vanilla ubuntu using the following command. The command is idempotent (convergent). Everything is pre-set, and the only thing you need to worry about is that the interface eth0 is 10Gbps-capable.
+
=== last 3 modified files ===
 +
  find . -type f -exec stat -c '%Y %n' {} \; | sort -nr | awk 'NR==1,NR==3 {print $2}'
  
−
ssh -i access/id_rsa_192_168_100_30 root@192.168.100.42 "rm -rfv /etc/chef" && \
+
=== Find 10 largest files in a folder ===
−
knife node delete pi_vm_multiccow_42 -y ; knife client delete pi_vm_multiccow_42 -y ; \
+
−
knife bootstrap 192.168.100.42 -N pi_vm_multiccow_42 -r "role[nedge-development]" -i access/id_rsa_192_168_100_30 -y -x root
+
  
−
If the interface is eth1:
+
  du -a /var | sort -n -r | head -n 10
 +
  du -a .    | sort -n -r | head -n 15
  
−
ssh -i access/id_rsa_192_168_100_30 root@192.168.100.42 "rm -rfv /etc/chef" && \
+
For mac:
−
knife node delete pi_vm_multiccow_42 -y ; knife client delete pi_vm_multiccow_42 -y ; \
+
−
knife bootstrap 192.168.100.42 -N pi_vm_multiccow_42 -r "role[nedge-development-eth1]" -i access/id_rsa_192_168_100_30 -y -x root
+
  
−
=== Nodes ===
+
du -a . | sort -n -r | head
  
−
This one, before I started sticking Node.js in there.
+
Limit how much logs journalctl keeps:
  
−
WARNING! in the codebase node listens on 3000. In production it listens on 3007.
+
  journalctl --vacuum-time=2d
 +
  journalctl --vacuum-size=500M
  
−
<pre>
+
=== Count Lines in a Folder ===
−
piousbox@piousbox-laptop:~/projects/rails-quick-start$ knife node show app_server_21
+
find . -name "*js" -print | grep -v node_modules | xargs cat | grep -v "^//" | grep -v "^$" | wc -l
−
Node Name:  app_server_21
+
−
Environment: _default
+
−
FQDN:       
+
−
IP:          10.0.1.151
+
−
Run List:    role[base], role[ish_static], role[marchesi_static], role[wiki_cac], role[wp_wasya]
+
−
Roles:      base, ish_static, marchesi_static, wiki_cac, wp_wasya
+
−
Recipes:    zsh, users::sysadmins, sudo, apt, nagios::client, git, build-essential, ish::base_apache, ish::static_site, ish_apache::static_site, ish_apache::base_apache, ish::base_php, mysql::client, mediawiki::default, mediawiki::appserver, wordpress::default
+
−
Platform:    ubuntu 12.04
+
−
Tags:     
+
−
</pre>
+
  
−
This is a successful deployment except change node's listening from 3000 to 3007:
 
−
<pre>
 
−
piousbox@piousbox-laptop:~/projects/rails-quick-start$ knife node show app_server_21
 
−
Node Name:  app_server_21
 
−
Environment: _default
 
−
FQDN:       
 
−
IP:          10.0.1.151
 
−
Run List:    role[base], role[node_exampler]
 
−
Roles:      base, node_exampler
 
−
Recipes:    zsh, users::sysadmins, sudo, apt, nagios::client, git, build-essential, node::base, node::upstream
 
−
Platform:    ubuntu 12.04
 
−
Tags:       
 
−
</pre>
 
  
−
The load balancer:
+
=== cut after ag for shorter lines ===
−
<pre>
+
−
ubuntu@ip-10-0-0-52:/etc/apache2/sites-available$ pwd
+
−
/etc/apache2/sites-available
+
−
ubuntu@ip-10-0-0-52:/etc/apache2/sites-available$ cat startups_staging
+
  
−
<VirtualHost *:80>
+
ag <your-search-keyword> | cut -c-800
−
  ServerAdmin piousbox@gmail.com
+
−
  ServerName annesque.com
+
−
  KeepAlive On
+
−
  ProxyVia On
+
−
  ProxyPreserveHost On
+
−
  ServerAlias staging.annesque.com
+
−
  ServerAlias startups.piousbox.com
+
−
  ServerAlias staging.startups.piousbox.com
+
−
  ProxyPreserveHost On
+
−
  ProxyVia On
+
−
  <proxy>
+
−
    Order deny,allow
+
−
    Allow from all
+
−
  </proxy>
+
−
  ProxyPass /api/ http://10.0.1.151:3007/ connectiontimeout=15 timeout=45
+
−
  ProxyPassReverse /api/ http://10.0.1.151:3007/
+
−
  ProxyPass / http://10.0.1.72:3010/ connectiontimeout=15 timeout=45
+
−
  ProxyPassReverse / http://10.0.1.72:3010/
+
−
</VirtualHost>
+
−
</pre>
+
  
−
=== Utils ===
+
=== increase number of open files, file descriptors ===
−
Remember the virtual site for combining a node backend into some other service?
+
  
 +
From: /etc/security/limits.conf
 
<pre>
 
<pre>
−
<VirtualHost *:3098>
+
* - nofile 131072
−
  ServerAdmin piousbox@gmail.com
+
 
−
  ServerName localhost
+
*    soft nofile 64000
−
  KeepAlive On
+
*    hard nofile 64000
−
  ProxyVia On
+
root soft nofile 64000
−
  ProxyPreserveHost On
+
root hard nofile 64000
−
  <proxy>
+
−
    Order deny,allow
+
−
    Allow from all
+
−
  </proxy>
+
−
  ProxyPass /api/ http://localhost:3000/ connectiontimeout=15 timeout=45
+
−
  ProxyPassReverse /api/ http://localhost:3000/
+
−
  ProxyPass / http://localhost:8000/ connectiontimeout=15 timeout=45
+
−
  ProxyPassReverse / http://localhost:8000/
+
−
</VirtualHost>
+
 
</pre>
 
</pre>
  
−
==== Establish a simple ssh port forward ====
+
Reload services after making changes, logout.
 +
 
 +
=== count open files ===
 +
lsof | wc -
 +
Per user:
 +
lsof | grep ' opera ' | awk '{print $NF}' | sort | wc -l
 +
 
 +
=== Set date ===
 +
sudo date --set "25 Sep 2013 15:00:00"
 +
 
 +
=== Get date in milliseconds ===
 +
date -d 2017-05-05 +%s
 +
 
 +
 
 +
 
 +
== Disks, Storage ==
 +
 
 +
=== check disks on ubuntu ===
 +
 
 +
  sudo lsblk
 +
 
 +
=== Mount a block device ===
 +
 
 +
From: https://www.digitalocean.com/community/tutorials/how-to-use-block-storage-on-digitalocean
 +
 
 +
  parted /dev/sda mklabel gpt
 +
  parted -a opt /dev/sda mkpart primary ext4 0% 100%
 +
  mkfs.ext4 /dev/sda
 +
  mkdir -p /data
 +
  '/dev/sda /data ext4 defaults,nofail,discard 0 2' | sudo tee -a /etc/fstab
 +
  mount -a
 +
 
 +
=== Find large files ===
 +
find / -xdev -type f -size +100M
 +
 
 +
=== see timestamps in history ===
 +
echo 'export HISTTIMEFORMAT="%d/%m/%y %T "' >> ~/.bashrc ; source ~/.bashrc
 +
 
 +
=== Generate large random file ===
 +
dd if=/dev/urandom of=/var/log/mongodb/randomFile bs=1M count=200
 +
 
 +
=== Add ssh key to a server ===
 +
cat ~/.ssh/id_rsa.pub | ssh user@hostname 'cat >> .ssh/authorized_keys'
 +
 
 +
=== Add identity to a server ===
 +
ssh-copy-id -i ~.ssh/key_name.pub user@host
 +
 
 +
=== Create a user ===
 +
 
 +
su - <username>
 +
sudo visudo -f /etc/sudoers
 +
usernameusedforlogin ALL=(ALL) NOPASSWD:ALL
 +
sudo adduser --disabled-password <username>
 +
 
 +
add user to group:
 +
 
 +
usermod -a -G wheel niceguy
 +
useradd -g wheel niceguy ## only is user doesn't exist!
 +
 
 +
or in rhel:
 +
 
 +
passwd -f -u deploy
 +
(and there is deluser)
 +
(I don't think I need this: usermod -aG sudo username ) - for making him a sudoer
 +
 
 +
useradd? adduser?
 +
 
 +
adduser : add user with full profile and info (pass, quota, permission, etc.) adduser is friendlier in that it sets up the account's home folders and other settings (e.g. automatically loading system stats and notifications on login),
 +
 
 +
whereas useradd just creates the user
 +
 
 +
=== Establish a simple ssh port forward ===
  
 
10.0.0.x is public, 10.0.1.x is private. Do:
 
10.0.0.x is public, 10.0.1.x is private. Do:
Line 110: Line 139:
 
  sudo iptables -t nat -A POSTROUTING -j MASQUERADE
 
  sudo iptables -t nat -A POSTROUTING -j MASQUERADE
  
−
=== Reset Chef UI Password ===
+
=== Delete files older than 5 days ===
−
  sudo -u opscode-pgsql /opt/chef-server/embedded/bin/psql opscode_chef
+
  find /path/to/files* -mtime +5 -exec rm {} \;
−
update osc_users set hashed_password = '$2a$12$y31Wno2MKiGXS3FSgVg5UunKG48gJz0pRV//RMy1osDxVbrb0On4W' , salt ='$2a$12$y31Wno2MKiGXS3FSgVg5Uu' where username ='admin';
+
  find /path/to/files* -type f -mtime +5 -exec rm {} -v \;
 +
 
 +
=== get children of a process ===
 +
pgrep -P $your_process1_pid
 +
 
 +
=== restrict resources given to a process ===
  
−
and login with user 'admin' and password 'password'.
+
sudo apt-get install cgroup-bin -y
 +
sudo cgcreate -g memory:/rmGroup
 +
sudo
 +
  echo $(( 500 * 1024 * 1024 )) > /sys/fs/cgroup/memory/rmGroup/memory.limit_in_bytes
 +
  # echo $(( 5000 * 1024 * 1024 )) > /sys/fs/cgroup/memory/rmGroup/memory.memsw.limit_in_bytes # swap only
 +
# cgexec -g memory:rmGroup <command>
 +
cgexec -g memory:rmGroup find cache/ -type f -mtime +10 -exec rm {} -v \;

Latest revision as of 17:00, 12 April 2024

Utils

Replace all occurrences of a string in a folder

ubuntu? 

find /home/user/ -type f | xargs sed -i  's/a.example.com/b.example.com/g'

mac: 

 LC_ALL=C find . -type f -name "*.html" -exec sed -i '' "s/UA-47088821-3/UA-47088821-4/" {} +

last 3 accessed files

 find . -type f -exec stat -c '%X %n' {} \; | sort -nr | awk 'NR==1,NR==3 {print $2}'

last 3 modified files

  find . -type f -exec stat -c '%Y %n' {} \; | sort -nr | awk 'NR==1,NR==3 {print $2}'

Find 10 largest files in a folder

 du -a /var | sort -n -r | head -n 10
 du -a .    | sort -n -r | head -n 15

For mac: 

du -a . | sort -n -r | head

Limit how much logs journalctl keeps: 

 journalctl --vacuum-time=2d
 journalctl --vacuum-size=500M

Count Lines in a Folder

find . -name "*js" -print | grep -v node_modules | xargs cat | grep -v "^//" | grep -v "^$" | wc -l


cut after ag for shorter lines

ag <your-search-keyword> | cut -c-800

increase number of open files, file descriptors

From: /etc/security/limits.conf 

* - nofile 131072

*    soft nofile 64000
*    hard nofile 64000
root soft nofile 64000
root hard nofile 64000

Reload services after making changes, logout.

count open files

lsof | wc -

Per user: 

lsof | grep ' opera ' | awk '{print $NF}' | sort | wc -l

Set date

sudo date --set "25 Sep 2013 15:00:00"

Get date in milliseconds

date -d 2017-05-05 +%s


Disks, Storage

check disks on ubuntu

 sudo lsblk

Mount a block device

From: https://www.digitalocean.com/community/tutorials/how-to-use-block-storage-on-digitalocean 

 parted /dev/sda mklabel gpt
 parted -a opt /dev/sda mkpart primary ext4 0% 100% 
 mkfs.ext4 /dev/sda
 mkdir -p /data
 '/dev/sda /data ext4 defaults,nofail,discard 0 2' | sudo tee -a /etc/fstab
 mount -a

Find large files

find / -xdev -type f -size +100M

see timestamps in history

echo 'export HISTTIMEFORMAT="%d/%m/%y %T "' >> ~/.bashrc ; source ~/.bashrc

Generate large random file

dd if=/dev/urandom of=/var/log/mongodb/randomFile bs=1M count=200

Add ssh key to a server

cat ~/.ssh/id_rsa.pub | ssh user@hostname 'cat >> .ssh/authorized_keys'

Add identity to a server

ssh-copy-id -i ~.ssh/key_name.pub user@host

Create a user

su - <username>
sudo visudo -f /etc/sudoers
usernameusedforlogin ALL=(ALL) NOPASSWD:ALL
sudo adduser --disabled-password <username>

add user to group: 

usermod -a -G wheel niceguy
useradd -g wheel niceguy ## only is user doesn't exist!

or in rhel: 

passwd -f -u deploy
(and there is deluser)
(I don't think I need this: usermod -aG sudo username ) - for making him a sudoer

useradd? adduser?

adduser : add user with full profile and info (pass, quota, permission, etc.) adduser is friendlier in that it sets up the account's home folders and other settings (e.g. automatically loading system stats and notifications on login),

whereas useradd just creates the user

Establish a simple ssh port forward

10.0.0.x is public, 10.0.1.x is private. Do: 

sudo sysctl net.ipv4.ip_forward=1
sudo iptables -t nat -A PREROUTING -p tcp -d 10.0.0.229 --dport 2270 -j DNAT --to-destination 10.0.1.224:22
sudo iptables -t nat -A POSTROUTING -j MASQUERADE

Delete files older than 5 days

find /path/to/files* -mtime +5 -exec rm {} \;
find /path/to/files* -type f -mtime +5 -exec rm {} -v \;

get children of a process

pgrep -P $your_process1_pid

restrict resources given to a process

sudo apt-get install cgroup-bin -y
sudo cgcreate -g memory:/rmGroup
sudo
 echo $(( 500 * 1024 * 1024 )) > /sys/fs/cgroup/memory/rmGroup/memory.limit_in_bytes
 # echo $(( 5000 * 1024 * 1024 )) > /sys/fs/cgroup/memory/rmGroup/memory.memsw.limit_in_bytes # swap only
# cgexec -g memory:rmGroup <command>
cgexec -g memory:rmGroup find cache/ -type f -mtime +10 -exec rm {} -v \;
Retrieved from "https://wiki.wasya.co/index.php?title=Linux_utils&oldid=3007"