| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-19 | |||
| 13:46:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add DownCellFixture https://review.openstack.org/614810 | |
| 14:03:09 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add API ref guideline for body text https://review.openstack.org/605628 | |
| 14:12:57 | openstackgerrit | Josephine Seifert proposed openstack/nova-specs master: Spec for the Nova part of Image Encryption https://review.openstack.org/608696 | |
| 14:32:24 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix server query examples https://review.openstack.org/616834 | |
| 14:56:38 | openstackgerrit | Elod Illes proposed openstack/nova master: Transform scheduler.select_destinations notification https://review.openstack.org/508506 | |
| 14:56:39 | mriedem | jackding: some small changes for https://review.openstack.org/#/c/605902/ and then i'm +2 | |
| 14:56:44 | mriedem | plus the glance image docs | |
| 15:04:03 | jackding | mriedem: ok thanks | |
| 15:04:27 | spatel | sean-k-mooney: morning!!! | |
| 15:19:17 | sean-k-mooney | hello | |
| 15:19:23 | spatel_ | how are you ? | |
| 15:20:02 | spatel_ | i had question related multi-queue for virt-io | |
| 15:20:06 | sean-k-mooney | not too bad. just rootcaused an issue in some logs so feeling quite good | |
| 15:21:00 | spatel_ | if i have 8 vCPU on instance in that queue what value should be good to set in ethtool -L eth0 combined <N> ? | |
| 15:22:09 | sean-k-mooney | 8 | |
| 15:22:37 | sean-k-mooney | when you enable multiqueue openstack will allocate a tx and rx pair per core | |
| 15:24:17 | spatel_ | i thought i should keep some CPU core for other work | |
| 15:24:45 | spatel_ | This is what i am doing so far may be i am wrong here.. if i create 16 vCPU machine then i assign only 8 queue | |
| 15:25:06 | spatel_ | if i create 8 vCPU machine then i create 4 queue | |
| 15:25:42 | spatel_ | am i doing something wrong doing that or best practice is to give vCPU = Queue size | |
| 15:26:07 | sean-k-mooney | assinging a queue does nto mean you are assinging a cpu to manage it | |
| 15:26:39 | sean-k-mooney | the vswitch cannot know which queue the os is using so it will assume it can enqueue packets to all of them | |
| 15:27:11 | sean-k-mooney | if you dont config the kernel to pool on all queue you will miss packets | |
| 15:27:36 | sean-k-mooney | you can have one guess cpu pool all queue if you want | |
| 15:28:12 | spatel_ | so best option is to give all queue right? | |
| 15:28:25 | spatel_ | 8 vCPU = 8 queue ? | |
| 15:28:31 | sean-k-mooney | by assingin 1 queue per cpus its allows apps to transmit without haveing to take locks as you can use the cpu id as a queue index | |
| 15:28:37 | sean-k-mooney | yes | |
| 15:29:03 | sean-k-mooney | but your application that is processign the traffic in the vm can be confied to only 4 cpus if you like | |
| 15:29:25 | sean-k-mooney | you just need to configre the app to use 2 queues per core in that case | |
| 15:29:48 | sean-k-mooney | if you are using the kernel driver this is all handeled internally i belive | |
| 15:30:42 | spatel_ | I thought queue bind with vCPU so if you specify 4 queue size then only 4 vCPU parellel work on packet processing.. | |
| 15:31:25 | spatel_ | that is why i was reserving 4 vCPU for OS and 4vCPU for queue (for networking work) | |
| 15:31:30 | sean-k-mooney | there are 2 level here. the host level were we will allocate 1 queu per vcpu and the guest level | |
| 15:32:02 | spatel_ | https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/libvirt-virtiomq.html | |
| 15:32:40 | spatel_ | Host level ?? never heard that | |
| 15:33:24 | sean-k-mooney | what im trying to say is that nova will configure qemu with 1 queu per vcpu form the host/hyperviour perspective | |
| 15:33:44 | sean-k-mooney | provided you enable multique that is | |
| 15:34:47 | spatel_ | hmm! | |
| 15:34:59 | sean-k-mooney | as such if you do not set ethtool -L <NIC> combined $(nproc) | |
| 15:35:08 | sean-k-mooney | you will get incorrect behavior | |
| 15:35:16 | tobias-urdin | mriedem: could i steal some of your time, perhaps you know if it flushed the system metadata cache | |
| 15:35:19 | tobias-urdin | 12:58 < tobias-urdin> is it possible to flush the entries in instance_system_metadata for an instance manually without breaking something? there is cached content on user and project which is incorrect | |
| 15:35:23 | tobias-urdin | 12:58 < tobias-urdin> i assume the only other wait to force a rewrite of those values in the xml would be a hard reboot, but not sure that would do the trick either | |
| 15:35:26 | spatel_ | i think slowly i am following you... what you trying to say.. sean-k-mooney | |
| 15:35:55 | sean-k-mooney | but within the guest the application that process packts form that interface can pin to 1-n core independly form the number of quese that there are | |
| 15:35:55 | openstackgerrit | Tobias Urdin proposed openstack/nova stable/rocky: Handle unicode characters in migration params https://review.openstack.org/618734 | |
| 15:36:38 | spatel_ | oh!! got it.. what you saying.. | |
| 15:36:58 | spatel_ | i was wrong about queuing logic... | |
| 15:37:20 | mriedem | tobias-urdin: there is no external REST API for changing things within the instance system_metadata, | |
| 15:37:27 | spatel_ | let me change my all instance to use all nproc | |
| 15:37:41 | mriedem | and what goes in there depends on the instance and operation, and manually messing with it is probably a bad idea given it's system data | |
| 15:38:04 | mriedem | tobias-urdin: so you'd have to be more specific about the incorrect sysmeta data that needs to be fixed | |
| 15:38:05 | spatel_ | sean-k-mooney: thanks again for saving me!!! | |
| 15:38:10 | tobias-urdin | mriedem: we are affected by https://review.openstack.org/#/c/618734/ but we have renamed the project until then but live migrations fails since it's wrong in the XML | |
| 15:38:31 | sean-k-mooney | spatel_: no worries this is rather difficult to figure out all at once | |
| 15:38:49 | tobias-urdin | so i was not sure if doing a hard reboot would use the cached instance_system_metadata or not, it does rebuild the XML but perhaps with the wrong data | |
| 15:38:59 | tobias-urdin | perhaps better to just test it out in a test environment | |
| 15:39:26 | spatel_ | in document is not very clear! anyway i am totally on it now | |
| 15:39:28 | tobias-urdin | it's probably not a good idea to DELETE those rows in instance_system_metadata | |
| 15:39:56 | mriedem | tobias-urdin: no that's probably not a great idea | |
| 15:40:25 | mriedem | tobias-urdin: what are the bad fields? user name and project name? | |
| 15:40:52 | mriedem | i.e. owner_user_name and owner_project_name in the xml? | |
| 15:41:14 | sean-k-mooney | spatel_: ya this was a topic at the summit. we need an nfv tuning guide with examples as the docs for these advanced feature are... lacking clarity to put it nicely | |
| 15:41:40 | tobias-urdin | mriedem: yeah the owner project name in the xml | |
| 15:42:02 | mriedem | i'm guessing _get_guest_config_meta in the libvirt driver is the thing putting the bad data in the xml | |
| 15:42:22 | mriedem | if you deleted owner_user_name and owner_project_name from the instance_system_metadata, we would put "N/A" in the xml | |
| 15:42:24 | sean-k-mooney | spatel_: i as a developer often use the specs to understand what we intended to enable but end users should not needed to read them to consume the features and use them properly. | |
| 15:43:44 | spatel_ | ++ | |
| 15:44:04 | mriedem | tobias-urdin: and those fields are put into the system_metadata only at server create time, not reboot | |
| 15:44:21 | mriedem | so even if you changed the user name / project name in keystone, that wouldn't be reflected in the instance sysmeta | |
| 15:46:33 | tobias-urdin | mriedem: thanks! then i'll apply the patch manually on the affected nodes until it lands in stable/rocky | |
| 15:52:17 | dansmith | mriedem: tobias-urdin +W | |
| 15:55:37 | dansmith | mriedem: jaypipes sean-k-mooney: I think we need to give some clarity to sundar here: https://review.openstack.org/#/c/603955/10/specs/stein/approved/nova-cyborg-interaction.rst | |
| 15:55:56 | dansmith | I am majorly opposed to documenting all the cyborg and os-acc APIs and functions in that spec | |
| 15:56:08 | sean-k-mooney | dansmith: just hopping on a call in 5 mins but sure what had you in mind | |
| 15:56:22 | sean-k-mooney | dansmith: yes i suggested splitting it into 3 specs a while ago | |
| 15:56:33 | dansmith | L324 specifically | |
| 15:56:54 | dansmith | sean-k-mooney: even if it gets split, those definitions don't belong in one of our specs, IMHO | |
| 15:57:57 | sean-k-mooney | am i highlevel discription of the os-acc that nova will consume would be nice but the cyborg apis dont need to be in the spec for sure | |
| 15:58:29 | sean-k-mooney | i would like to know the data structure that are passed the the fuction that should be called at diffrent point by nova but the internal can live in cyborg specs | |
| 15:59:19 | dansmith | sean-k-mooney: the data structures are fine, but you asked him to add more api signatures where I asked him to completely remove them | |
| 15:59:25 | sean-k-mooney | dansmith: i think the os-vif spec got it about right https://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/os-vif-library.html | |
| 15:59:27 | dansmith | which is why we need to converge so he knows what to do | |
| 15:59:51 | jaypipes | device plugging lib. | |
| 15:59:51 | jaypipes | sean-k-mooney: I think I was pretty clear in Denver and elsewhere that I think the Cyborg team first needs to have some workable low-level device plumbing library (what I *thought* os-acc was supposed to be) before Nova even becomes part of the conversation. I view os-acc as very similar to os-vif, and I'm disappointed that there is even discussion about REST APIs or anything like that instead of designing an easy to use, hard-to-abuse low-level | |
| 16:00:25 | sean-k-mooney | dansmith: well that was because he was documenting the api and did not discribe the device profile enpoint | |
| 16:00:38 | sean-k-mooney | so either remove the api enpoitn entirely which im fine with or include that one | |
| 16:00:52 | sean-k-mooney | jaypipes: yes i agree with that | |
| 16:00:55 | jaypipes | sean-k-mooney: totally unnecessary at this point, IMHO. | |
| 16:01:03 | jaypipes | sean-k-mooney: (the REST API or anything like that) | |
| 16:01:29 | sean-k-mooney | jaypipes: ya so let ask for all the cyborg rest apis to be removed and state it will be handeled by os-acc | |
| 16:01:32 | dansmith | jaypipes: can you comment on the spec (L324ish)? I've already voiced there and so another "remove all this" would be good.. | |
| 16:01:43 | jaypipes | sean-k-mooney: the path to success with Cyborg is to mimick os-vif, not Mogan. | |
| 16:01:54 | sean-k-mooney | jaypipes: yes i agree | |
| 16:02:04 | sean-k-mooney | jaypipes: anyway i need to jump on a call | |
| 16:02:08 | jaypipes | dansmith: ok. I've already made a number of those comments but will do so again | |
| 16:02:30 | dansmith | jaypipes: I know, and I appreciate it.. L324 is where there was a conflicting "Do X" and "Don't do X" | |
| 16:02:41 | sean-k-mooney | dansmith: jaypipes feel free to link to this irc convo or just state im +1 on removing the api enpoint too | |
| 16:02:45 | jaypipes | dansmith: ack | |