Difference between revisions of "Linux utils"

From Wasya Wiki
Jump to: navigation, search
(Create a user)
(Disks, Storage)
 
(30 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
= Utils =
 
= Utils =
 +
* [[ css ]]
 +
 
* [[ Git cheatsheet ]]
 
* [[ Git cheatsheet ]]
 
* [[ IO Utils ]]
 
* [[ IO Utils ]]
 
* [[ json ]]
 
* [[ json ]]
 +
* [[ kubernetes ]]
 
* [[ Memory Utils ]]
 
* [[ Memory Utils ]]
* [[ Networking ]]
+
 
 
* [[ Network Utils ]]
 
* [[ Network Utils ]]
 
* [[ Process Utils ]]
 
* [[ Process Utils ]]
Line 16: Line 19:
 
  LC_ALL=C find . -type f -name "*.html" -exec sed -i '' "s/UA-47088821-3/UA-47088821-4/" {} +
 
  LC_ALL=C find . -type f -name "*.html" -exec sed -i '' "s/UA-47088821-3/UA-47088821-4/" {} +
 
</pre>
 
</pre>
 +
 +
=== last 3 accessed files ===
 +
  find . -type f -exec stat -c '%X %n' {} \; | sort -nr | awk 'NR==1,NR==3 {print $2}'
 +
 +
=== last 3 modified files ===
 +
  find . -type f -exec stat -c '%Y %n' {} \; | sort -nr | awk 'NR==1,NR==3 {print $2}'
  
 
=== Find 10 largest files in a folder ===
 
=== Find 10 largest files in a folder ===
du -a /var | sort -n -r | head -n 10
+
 
 +
  du -a /var | sort -n -r | head -n 10
 +
  du -a .    | sort -n -r | head -n 15
 +
 
 +
For mac:
 +
 
 +
du -a . | sort -n -r | head
 +
 
 +
Limit how much logs journalctl keeps:
 +
 
 +
  journalctl --vacuum-time=2d
 +
  journalctl --vacuum-size=500M
  
 
=== Count Lines in a Folder ===
 
=== Count Lines in a Folder ===
 
  find . -name "*js" -print | grep -v node_modules | xargs cat | grep -v "^//" | grep -v "^$" | wc -l
 
  find . -name "*js" -print | grep -v node_modules | xargs cat | grep -v "^//" | grep -v "^$" | wc -l
  
== System ==
+
 
 +
=== cut after ag for shorter lines ===
 +
 
 +
ag <your-search-keyword> | cut -c-800
 +
 
 +
=== increase number of open files, file descriptors ===
 +
 
 +
From: /etc/security/limits.conf
 +
<pre>
 +
* - nofile 131072
 +
 
 +
*    soft nofile 64000
 +
*    hard nofile 64000
 +
root soft nofile 64000
 +
root hard nofile 64000
 +
</pre>
 +
 
 +
Reload services after making changes, logout.
 +
 
 +
=== count open files ===
 +
lsof | wc -
 +
Per user:
 +
lsof | grep ' opera ' | awk '{print $NF}' | sort | wc -l
 +
 
 
=== Set date ===
 
=== Set date ===
 
  sudo date --set "25 Sep 2013 15:00:00"
 
  sudo date --set "25 Sep 2013 15:00:00"
 +
 
=== Get date in milliseconds ===
 
=== Get date in milliseconds ===
 
  date -d 2017-05-05 +%s
 
  date -d 2017-05-05 +%s
 +
 +
  
 
== Disks, Storage ==
 
== Disks, Storage ==
 +
 
=== check disks on ubuntu ===  
 
=== check disks on ubuntu ===  
sudo lsblk
 
=== Mount a block devise ===
 
From: https://www.digitalocean.com/community/tutorials/how-to-use-block-storage-on-digitalocean
 
parted /dev/sda mklabel gpt
 
parted -a opt /dev/sda mkpart primary ext4 0% 100%
 
mkfs.ext4 /dev/sda
 
mkdir -p /data
 
'/dev/sda /data ext4 defaults,nofail,discard 0 2' | sudo tee -a /etc/fstab
 
mount -a
 
  
 +
  sudo lsblk
  
 +
=== Mount a block device ===
 +
 +
From: https://www.digitalocean.com/community/tutorials/how-to-use-block-storage-on-digitalocean
  
=== ubuntu vpn ===
+
  parted /dev/sda mklabel gpt
https://hide.me/en/vpnsetup/ubuntu/openvpn/
+
  parted -a opt /dev/sda mkpart primary ext4 0% 100%
 +
  mkfs.ext4 /dev/sda
 +
  mkdir -p /data
 +
  '/dev/sda /data ext4 defaults,nofail,discard 0 2' | sudo tee -a /etc/fstab
 +
  mount -a
  
 
=== Find large files ===
 
=== Find large files ===
Line 55: Line 101:
 
  dd if=/dev/urandom of=/var/log/mongodb/randomFile bs=1M count=200
 
  dd if=/dev/urandom of=/var/log/mongodb/randomFile bs=1M count=200
  
=== Add ssh key to a server ===
 
cat ~/.ssh/id_rsa.pub | ssh user@hostname 'cat >> .ssh/authorized_keys'
 
 
=== Add identity to a server ===
 
ssh-copy-id -i ~.ssh/key_name.pub user@host
 
  
 
=== Create a user ===
 
=== Create a user ===
useradd? adduser?
 
 
adduser : add user with full profile and info (pass, quota, permission, etc.) adduser is friendlier in that it sets up the account's home folders and other settings (e.g. automatically loading system stats and notifications on login),
 
 
whereas useradd just creates the user
 
  
 
  su - <username>
 
  su - <username>
Line 72: Line 108:
 
  usernameusedforlogin ALL=(ALL) NOPASSWD:ALL
 
  usernameusedforlogin ALL=(ALL) NOPASSWD:ALL
 
  sudo adduser --disabled-password <username>
 
  sudo adduser --disabled-password <username>
 +
 +
add user to group:
 +
 +
usermod -a -G wheel niceguy
 +
useradd -g wheel niceguy ## only is user doesn't exist!
 +
 +
or in rhel:
 +
 +
passwd -f -u deploy
 
  (and there is deluser)
 
  (and there is deluser)
 
  (I don't think I need this: usermod -aG sudo username ) - for making him a sudoer
 
  (I don't think I need this: usermod -aG sudo username ) - for making him a sudoer
  
=== Establish a simple ssh port forward ===
+
useradd? adduser?
  
10.0.0.x is public, 10.0.1.x is private. Do:
+
adduser : add user with full profile and info (pass, quota, permission, etc.) adduser is friendlier in that it sets up the account's home folders and other settings (e.g. automatically loading system stats and notifications on login),
  
sudo sysctl net.ipv4.ip_forward=1
+
whereas useradd just creates the user
sudo iptables -t nat -A PREROUTING -p tcp -d 10.0.0.229 --dport 2270 -j DNAT --to-destination 10.0.1.224:22
+
sudo iptables -t nat -A POSTROUTING -j MASQUERADE
+
  
=== Reset Chef UI Password ===
 
sudo -u opscode-pgsql  /opt/chef-server/embedded/bin/psql opscode_chef
 
update osc_users set hashed_password = '$2a$12$y31Wno2MKiGXS3FSgVg5UunKG48gJz0pRV//RMy1osDxVbrb0On4W' , salt ='$2a$12$y31Wno2MKiGXS3FSgVg5Uu' where username ='admin';
 
  
and login with user 'admin' and password 'password'.
 
  
 
=== Delete files older than 5 days ===
 
=== Delete files older than 5 days ===
Line 105: Line 144:
 
  # cgexec -g memory:rmGroup <command>
 
  # cgexec -g memory:rmGroup <command>
 
  cgexec -g memory:rmGroup find cache/ -type f -mtime +10 -exec rm {} -v \;
 
  cgexec -g memory:rmGroup find cache/ -type f -mtime +10 -exec rm {} -v \;
 
== System ==
 
=== increase number of open files, file descriptors ===
 
 
From: /etc/security/limits.conf
 
<pre>
 
* - nofile 131072
 
 
*    soft nofile 64000
 
*    hard nofile 64000
 
root soft nofile 64000
 
root hard nofile 64000
 
</pre>
 
 
Reload services after making changes, logout.
 
=== count open files ===
 
lsof | wc -
 
Per user:
 
lsof | grep ' opera ' | awk '{print $NF}' | sort | wc -l
 

Latest revision as of 19:11, 29 September 2024

Utils

Replace all occurrences of a string in a folder

ubuntu?

find /home/user/ -type f | xargs sed -i  's/a.example.com/b.example.com/g'

mac:

 LC_ALL=C find . -type f -name "*.html" -exec sed -i '' "s/UA-47088821-3/UA-47088821-4/" {} +

last 3 accessed files

 find . -type f -exec stat -c '%X %n' {} \; | sort -nr | awk 'NR==1,NR==3 {print $2}'

last 3 modified files

  find . -type f -exec stat -c '%Y %n' {} \; | sort -nr | awk 'NR==1,NR==3 {print $2}'

Find 10 largest files in a folder

 du -a /var | sort -n -r | head -n 10
 du -a .    | sort -n -r | head -n 15

For mac:

du -a . | sort -n -r | head

Limit how much logs journalctl keeps:

 journalctl --vacuum-time=2d
 journalctl --vacuum-size=500M

Count Lines in a Folder

find . -name "*js" -print | grep -v node_modules | xargs cat | grep -v "^//" | grep -v "^$" | wc -l


cut after ag for shorter lines

ag <your-search-keyword> | cut -c-800

increase number of open files, file descriptors

From: /etc/security/limits.conf

* - nofile 131072

*    soft nofile 64000
*    hard nofile 64000
root soft nofile 64000
root hard nofile 64000

Reload services after making changes, logout.

count open files

lsof | wc -

Per user:

lsof | grep ' opera ' | awk '{print $NF}' | sort | wc -l

Set date

sudo date --set "25 Sep 2013 15:00:00"

Get date in milliseconds

date -d 2017-05-05 +%s


Disks, Storage

check disks on ubuntu

 sudo lsblk

Mount a block device

From: https://www.digitalocean.com/community/tutorials/how-to-use-block-storage-on-digitalocean

 parted /dev/sda mklabel gpt
 parted -a opt /dev/sda mkpart primary ext4 0% 100% 
 mkfs.ext4 /dev/sda
 mkdir -p /data
 '/dev/sda /data ext4 defaults,nofail,discard 0 2' | sudo tee -a /etc/fstab
 mount -a

Find large files

find / -xdev -type f -size +100M

see timestamps in history

echo 'export HISTTIMEFORMAT="%d/%m/%y %T "' >> ~/.bashrc ; source ~/.bashrc

Generate large random file

dd if=/dev/urandom of=/var/log/mongodb/randomFile bs=1M count=200


Create a user

su - <username>
sudo visudo -f /etc/sudoers
usernameusedforlogin ALL=(ALL) NOPASSWD:ALL
sudo adduser --disabled-password <username>

add user to group:

usermod -a -G wheel niceguy
useradd -g wheel niceguy ## only is user doesn't exist!

or in rhel:

passwd -f -u deploy
(and there is deluser)
(I don't think I need this: usermod -aG sudo username ) - for making him a sudoer

useradd? adduser?

adduser : add user with full profile and info (pass, quota, permission, etc.) adduser is friendlier in that it sets up the account's home folders and other settings (e.g. automatically loading system stats and notifications on login),

whereas useradd just creates the user


Delete files older than 5 days

find /path/to/files* -mtime +5 -exec rm {} \;
find /path/to/files* -type f -mtime +5 -exec rm {} -v \;

get children of a process

pgrep -P $your_process1_pid

restrict resources given to a process

sudo apt-get install cgroup-bin -y
sudo cgcreate -g memory:/rmGroup
sudo
 echo $(( 500 * 1024 * 1024 )) > /sys/fs/cgroup/memory/rmGroup/memory.limit_in_bytes
 # echo $(( 5000 * 1024 * 1024 )) > /sys/fs/cgroup/memory/rmGroup/memory.memsw.limit_in_bytes # swap only
# cgexec -g memory:rmGroup <command>
cgexec -g memory:rmGroup find cache/ -type f -mtime +10 -exec rm {} -v \;