Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-11-19
09:36:36 kashyap melwitt: Have fun; and try to disconnect :-)
09:37:12 jsheeren we noticed the latency on that volume becoming very high and we were getting api timeouts. (with rabbitmq time-outs)
09:39:21 jsheeren our suspicion is the image cache manager as a cause of the latencies. when we did a rolling restart of nova-compute services (with a 5 min delay between them); the latency was gone
09:39:30 jsheeren and we are not seeing the api time-outs any more.
09:40:40 jsheeren has anyone had a similar experience? we increased the image cache manager interval as well, from the default 2400 sec to 4800 sec
09:43:16 jsheeren it seems the crawling of the /var/lib/nova tree takes longer than the message time-out. so we also increased the rcp_response time-out
09:43:43 jsheeren there are approx 700 instances on the env btw
09:55:47 openstackgerrit Matthew Booth proposed openstack/nova master: Call migrate_volume_completion after cinder migrate https://review.openstack.org/618717
10:39:46 tobias-urdin melwitt: we did hit this bug in rocky today https://review.openstack.org/#/c/591080/
10:39:53 tobias-urdin would that be acceptable for backport?
11:11:48 openstackgerrit Vlad Gusev proposed openstack/nova stable/rocky: Handle unicode characters in migration params https://review.openstack.org/618734
11:20:14 openstackgerrit Radoslav Gerganov proposed openstack/nova master: VMware: implement trigger crash dump https://review.openstack.org/618736
11:25:22 openstackgerrit Jason SUN proposed openstack/nova master: For instances end up in cell0, set its availability_zone to None. https://review.openstack.org/618737
12:09:34 tobias-urdin is it possible to flush the entries in instance_system_metadata for an instance manually without breaking something? there is cached content on user and project which is incorrect
12:09:53 tobias-urdin i assume the only other wait to force a rewrite of those values in the xml would be a hard reboot, but not sure that would do the trick either
13:25:07 openstackgerrit Surya Seetharaman proposed openstack/nova master: Add os_compute_api:servers:create:cell_down policy https://review.openstack.org/614783
13:41:20 openstackgerrit Surya Seetharaman proposed openstack/nova master: Add os_compute_api:servers:create:cell_down policy https://review.openstack.org/614783
13:46:11 openstackgerrit Matt Riedemann proposed openstack/nova master: Add DownCellFixture https://review.openstack.org/614810
14:03:09 openstackgerrit Takashi NATSUME proposed openstack/nova master: Add API ref guideline for body text https://review.openstack.org/605628
14:12:57 openstackgerrit Josephine Seifert proposed openstack/nova-specs master: Spec for the Nova part of Image Encryption https://review.openstack.org/608696
14:32:24 openstackgerrit Takashi NATSUME proposed openstack/nova master: Fix server query examples https://review.openstack.org/616834
14:56:38 openstackgerrit Elod Illes proposed openstack/nova master: Transform scheduler.select_destinations notification https://review.openstack.org/508506
14:56:39 mriedem jackding: some small changes for https://review.openstack.org/#/c/605902/ and then i'm +2
14:56:44 mriedem plus the glance image docs
15:04:03 jackding mriedem: ok thanks
15:04:27 spatel sean-k-mooney: morning!!!
15:19:17 sean-k-mooney hello
15:19:23 spatel_ how are you ?
15:20:02 spatel_ i had question related multi-queue for virt-io
15:20:06 sean-k-mooney not too bad. just rootcaused an issue in some logs so feeling quite good
15:21:00 spatel_ if i have 8 vCPU on instance in that queue what value should be good to set in ethtool -L eth0 combined <N> ?
15:22:09 sean-k-mooney 8
15:22:37 sean-k-mooney when you enable multiqueue openstack will allocate a tx and rx pair per core
15:24:17 spatel_ i thought i should keep some CPU core for other work
15:24:45 spatel_ This is what i am doing so far may be i am wrong here.. if i create 16 vCPU machine then i assign only 8 queue
15:25:06 spatel_ if i create 8 vCPU machine then i create 4 queue
15:25:42 spatel_ am i doing something wrong doing that or best practice is to give vCPU = Queue size
15:26:07 sean-k-mooney assinging a queue does nto mean you are assinging a cpu to manage it
15:26:39 sean-k-mooney the vswitch cannot know which queue the os is using so it will assume it can enqueue packets to all of them
15:27:11 sean-k-mooney if you dont config the kernel to pool on all queue you will miss packets
15:27:36 sean-k-mooney you can have one guess cpu pool all queue if you want
15:28:12 spatel_ so best option is to give all queue right?
15:28:25 spatel_ 8 vCPU = 8 queue ?
15:28:31 sean-k-mooney by assingin 1 queue per cpus its allows apps to transmit without haveing to take locks as you can use the cpu id as a queue index
15:28:37 sean-k-mooney yes
15:29:03 sean-k-mooney but your application that is processign the traffic in the vm can be confied to only 4 cpus if you like
15:29:25 sean-k-mooney you just need to configre the app to use 2 queues per core in that case
15:29:48 sean-k-mooney if you are using the kernel driver this is all handeled internally i belive
15:30:42 spatel_ I thought queue bind with vCPU so if you specify 4 queue size then only 4 vCPU parellel work on packet processing..
15:31:25 spatel_ that is why i was reserving 4 vCPU for OS and 4vCPU for queue (for networking work)
15:31:30 sean-k-mooney there are 2 level here. the host level were we will allocate 1 queu per vcpu and the guest level
15:32:02 spatel_ https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/libvirt-virtiomq.html
15:32:40 spatel_ Host level ?? never heard that
15:33:24 sean-k-mooney what im trying to say is that nova will configure qemu with 1 queu per vcpu form the host/hyperviour perspective
15:33:44 sean-k-mooney provided you enable multique that is
15:34:47 spatel_ hmm!
15:34:59 sean-k-mooney as such if you do not set ethtool -L <NIC> combined $(nproc)
15:35:08 sean-k-mooney you will get incorrect behavior
15:35:16 tobias-urdin mriedem: could i steal some of your time, perhaps you know if it flushed the system metadata cache
15:35:19 tobias-urdin 12:58 < tobias-urdin> is it possible to flush the entries in instance_system_metadata for an instance manually without breaking something? there is cached content on user and project which is incorrect
15:35:23 tobias-urdin 12:58 < tobias-urdin> i assume the only other wait to force a rewrite of those values in the xml would be a hard reboot, but not sure that would do the trick either
15:35:26 spatel_ i think slowly i am following you... what you trying to say.. sean-k-mooney
15:35:55 openstackgerrit Tobias Urdin proposed openstack/nova stable/rocky: Handle unicode characters in migration params https://review.openstack.org/618734
15:35:55 sean-k-mooney but within the guest the application that process packts form that interface can pin to 1-n core independly form the number of quese that there are
15:36:38 spatel_ oh!! got it.. what you saying..
15:36:58 spatel_ i was wrong about queuing logic...
15:37:20 mriedem tobias-urdin: there is no external REST API for changing things within the instance system_metadata,
15:37:27 spatel_ let me change my all instance to use all nproc
15:37:41 mriedem and what goes in there depends on the instance and operation, and manually messing with it is probably a bad idea given it's system data
15:38:04 mriedem tobias-urdin: so you'd have to be more specific about the incorrect sysmeta data that needs to be fixed
15:38:05 spatel_ sean-k-mooney: thanks again for saving me!!!
15:38:10 tobias-urdin mriedem: we are affected by https://review.openstack.org/#/c/618734/ but we have renamed the project until then but live migrations fails since it's wrong in the XML
15:38:31 sean-k-mooney spatel_: no worries this is rather difficult to figure out all at once
15:38:49 tobias-urdin so i was not sure if doing a hard reboot would use the cached instance_system_metadata or not, it does rebuild the XML but perhaps with the wrong data
15:38:59 tobias-urdin perhaps better to just test it out in a test environment
15:39:26 spatel_ in document is not very clear! anyway i am totally on it now
15:39:28 tobias-urdin it's probably not a good idea to DELETE those rows in instance_system_metadata
15:39:56 mriedem tobias-urdin: no that's probably not a great idea
15:40:25 mriedem tobias-urdin: what are the bad fields? user name and project name?
15:40:52 mriedem i.e. owner_user_name and owner_project_name in the xml?
15:41:14 sean-k-mooney spatel_: ya this was a topic at the summit. we need an nfv tuning guide with examples as the docs for these advanced feature are... lacking clarity to put it nicely
15:41:40 tobias-urdin mriedem: yeah the owner project name in the xml
15:42:02 mriedem i'm guessing _get_guest_config_meta in the libvirt driver is the thing putting the bad data in the xml
15:42:22 mriedem if you deleted owner_user_name and owner_project_name from the instance_system_metadata, we would put "N/A" in the xml
15:42:24 sean-k-mooney spatel_: i as a developer often use the specs to understand what we intended to enable but end users should not needed to read them to consume the features and use them properly.
15:43:44 spatel_ ++
15:44:04 mriedem tobias-urdin: and those fields are put into the system_metadata only at server create time, not reboot
15:44:21 mriedem so even if you changed the user name / project name in keystone, that wouldn't be reflected in the instance sysmeta
15:46:33 tobias-urdin mriedem: thanks! then i'll apply the patch manually on the affected nodes until it lands in stable/rocky
15:52:17 dansmith mriedem: tobias-urdin +W
15:55:37 dansmith mriedem: jaypipes sean-k-mooney: I think we need to give some clarity to sundar here: https://review.openstack.org/#/c/603955/10/specs/stein/approved/nova-cyborg-interaction.rst
15:55:56 dansmith I am majorly opposed to documenting all the cyborg and os-acc APIs and functions in that spec
15:56:08 sean-k-mooney dansmith: just hopping on a call in 5 mins but sure what had you in mind
15:56:22 sean-k-mooney dansmith: yes i suggested splitting it into 3 specs a while ago
15:56:33 dansmith L324 specifically
15:56:54 dansmith sean-k-mooney: even if it gets split, those definitions don't belong in one of our specs, IMHO
15:57:57 sean-k-mooney am i highlevel discription of the os-acc that nova will consume would be nice but the cyborg apis dont need to be in the spec for sure
15:58:29 sean-k-mooney i would like to know the data structure that are passed the the fuction that should be called at diffrent point by nova but the internal can live in cyborg specs
15:59:19 dansmith sean-k-mooney: the data structures are fine, but you asked him to add more api signatures where I asked him to completely remove them
15:59:25 sean-k-mooney dansmith: i think the os-vif spec got it about right https://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/os-vif-library.html

Earlier   Later