| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-11-20 | |||
| 07:18:24 | openstackgerrit | Radoslav Gerganov proposed openstack/nova master: VMware: implement trigger crash dump https://review.openstack.org/618736 | |
| 08:29:44 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add os_compute_api:servers:create:cell_down policy https://review.openstack.org/614783 | |
| 10:30:13 | sean-k-mooney | o/ | |
| 10:33:25 | openstackgerrit | Chris Dent proposed openstack/nova master: Use external placement in functional tests https://review.openstack.org/617941 | |
| 10:33:39 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Delete the placement code https://review.openstack.org/618215 | |
| 11:29:57 | openstackgerrit | Merged openstack/nova master: Nix refs to ResourceProvider obj from libvirt UT https://review.openstack.org/618786 | |
| 11:32:29 | cdent | huzzah | |
| 11:33:21 | sean-k-mooney | was the the last usage of placement object in the nova unit tests | |
| 11:34:03 | cdent | sean-k-mooney: not quite | |
| 11:34:14 | cdent | well, strictly speaking, yes | |
| 11:34:19 | cdent | it was the last usage of the objcts | |
| 11:34:32 | cdent | but there are some other tests which are testing things that want to use the placement db | |
| 11:34:37 | sean-k-mooney | but not the last useage of placement | |
| 11:34:42 | sean-k-mooney | ah ok | |
| 11:34:48 | cdent | in the my wip to remove the placement code I've had to remove some other tests/code | |
| 11:35:32 | cdent | when I get mat's suggestions on https://review.openstack.org/#/c/600161/ done, I'm going to go back to https://review.openstack.org/#/c/618215/ to fix up the merge conflict and tidy up whatever else is broken | |
| 11:40:00 | prometheanfire | http://logs.openstack.org/34/618834/1/check/cross-nova-py27/c3128e0/testr_results.html.gz new oslo.service causes failures | |
| 11:40:07 | prometheanfire | since people seem here :D | |
| 11:40:40 | openstackgerrit | Martin Midolesov proposed openstack/nova master: Allow driver to specify switch&port for faster lookup https://review.openstack.org/617695 | |
| 11:45:29 | cdent | prometheanfire: oh joy. I seem to recall efried and melwitt being vaguely aware of that | |
| 11:46:02 | sean-k-mooney | we were aware of that because we backported a fix downstream that we sould not have | |
| 11:46:44 | sean-k-mooney | prometheanfire: what branch is this happening on | |
| 11:47:28 | sean-k-mooney | prometheanfire: ya rocky nova is not compatible with that version of oslo service | |
| 11:47:58 | sean-k-mooney | we should not be increaseing the upper constatin on olso service in stable rocky anyway | |
| 11:48:37 | sean-k-mooney | oh it got backported into 1.31.6 ... | |
| 11:54:03 | prometheanfire | yep :| | |
| 11:54:21 | sean-k-mooney | https://review.openstack.org/#/c/616505/3 is the issue | |
| 11:55:00 | sean-k-mooney | or rather https://review.openstack.org/#/q/I62e9f1a7cde8846be368fbec58b8e0825ce02079 | |
| 11:55:12 | sean-k-mooney | well i guess its the same | |
| 12:10:46 | xek | did anyone try to use nova notifications lately? | |
| 12:10:51 | xek | I have an issue that when an instance is deleted, I only get an update notification with task state "deleting", but no instance.delete.end notification... | |
| 12:11:08 | xek | I tried both versioned and unversioned notifications | |
| 12:15:33 | jaosorior | gibi: are you around? | |
| 12:16:00 | jaosorior | gibi: I remember you had around some document where you had the new supported versioned notifications. | |
| 12:16:32 | xek | jaosorior, I think it's this one: https://docs.openstack.org/nova/latest/reference/notifications.html | |
| 12:17:24 | jaosorior | xek: instance.delete.end is there in the list | |
| 12:17:31 | jaosorior | so, if the notification is not being emmited, it's a bug. | |
| 12:18:05 | sean-k-mooney | xek: going froward i belive we are droping support for unverioned notifications. i dont belive we intend to remvoed them but we had discussed freezeing the code and not fixing new buts | |
| 12:18:08 | sean-k-mooney | *bugs | |
| 12:19:33 | jaosorior | sean-k-mooney: thanks, we're aware of the unversioned notifications deprecation. Hence why we started testing the versioned ones. instance.delete.end should be in the versioned notifications too (according to the list in the doc xek pointed at). So... if it's not being emmited, I think it's a bug. We'll file it up. | |
| 12:19:39 | jaosorior | unless that doc is outdated :/ | |
| 12:20:09 | sean-k-mooney | jaosorior: if the versioned notifcation is not beeing emited its a bug yes | |
| 12:20:45 | sean-k-mooney | that said the notifacion may have been lost in rabbitmq if you dont have perstency | |
| 12:20:51 | sean-k-mooney | i assume its repeatable | |
| 12:21:02 | jaosorior | sean-k-mooney: can yo ellaborate on that? | |
| 12:21:57 | sean-k-mooney | jaosorior: you can configure rabbitmq to persist each message to disk until its dequeued or it can keep them just in memory | |
| 12:22:26 | sean-k-mooney | using durable queue is a significant perfromace hit but if rabbit is restarted messages are not lost | |
| 12:22:40 | sean-k-mooney | i do not belive the queue used for notifcation are durabel | |
| 12:23:21 | jaosorior | sean-k-mooney: so, a bit of context: xek is working on getting functional tests for novajoin (a vendordata plugin for nova). These functional tests are being set up on top of devstack.... do you happen to know if this would be an issue that would hit a small devstack deployment? | |
| 12:23:47 | xek | sean-k-mooney, we saw some heartbeat misses and thus connectivity issues in the logs, but we didn't restart rabbitmq | |
| 12:23:54 | jaosorior | sean-k-mooney: we ultimately use it in TripleO, however, we haven't actually had issues with the notifications (yet). So I guess the rabbitmq settings are OK :D | |
| 12:23:58 | xek | sean-k-mooney, and it's pretty consistent | |
| 12:24:07 | sean-k-mooney | no idea unfortunetly i dont know what the default is | |
| 12:24:28 | xek | the default is non-persistent | |
| 12:24:45 | sean-k-mooney | oslo messaging and rabit by default do not gurantee message delivery as far as i am aware | |
| 12:24:59 | sean-k-mooney | if there was a tempary network issue the notifcaion could have been lost | |
| 12:26:11 | sean-k-mooney | my understanding is oslo messaging will not buffer the notifaction to my knolage and retry sendign the notification to rabbit if it trys to send it and fail unless you code that on top yourself | |
| 12:27:39 | sean-k-mooney | i could be wrong about that however as this is a part of nova i have realy dealt with | |
| 12:28:25 | jaosorior | sean-k-mooney: got it, I guess the next step is to investigate how rabbitmq is being configured in devstack, and how to modify that configuration if necessary | |
| 12:28:31 | jaosorior | sean-k-mooney: thanks for the guidance! | |
| 12:35:51 | openstackgerrit | Chris Dent proposed openstack/nova master: Use external placement in functional tests https://review.openstack.org/617941 | |
| 12:35:52 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Delete the placement code https://review.openstack.org/618215 | |
| 12:36:43 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Delete the placement code https://review.openstack.org/618215 | |
| 12:36:54 | sean-k-mooney | jaosorior: xek for what its worth it looks like you can enable durable queues in the nova.conf and it default to false | |
| 12:36:56 | sean-k-mooney | https://docs.openstack.org/oslo.messaging/ocata/opts.html#oslo_messaging_rabbit.amqp_durable_queues | |
| 12:37:34 | sean-k-mooney | i should proably check the latest doc actully to see if that is still a thing | |
| 12:38:11 | jangutter | sean-k-mooney: would you perhaps be able to donate some of your copious free time towards the helping with the naming of things? | |
| 12:39:02 | jaosorior | sean-k-mooney: thanks! we'll check that out. | |
| 12:39:37 | sean-k-mooney | jangutter: hah i could stop procrastinating by looking at messaging stuff and look at nameing instead | |
| 12:40:11 | sean-k-mooney | jaosorior: https://docs.openstack.org/oslo.messaging/latest/configuration/opts.html#oslo_messaging_rabbit.amqp_durable_queues its there on latest too | |
| 12:40:13 | jangutter | sean-k-mooney: bring your bikeshed, it can be any colour as long as it's red. | |
| 12:40:58 | sean-k-mooney | haha | |
| 12:41:04 | sean-k-mooney | not pink | |
| 12:41:14 | jangutter | sean-k-mooney: I think we're close to peak confusion with what the heck "network offloads" mean. | |
| 12:41:16 | sean-k-mooney | jangutter: is this related to the os-vif spec | |
| 12:41:24 | sean-k-mooney | ah yes | |
| 12:42:16 | jangutter | sean-k-mooney: yaas.... anyone wanting to frighten their young'uns: https://review.openstack.org/#/c/607610/ | |
| 12:42:58 | sean-k-mooney | jay highlighted it to me yesterday | |
| 12:43:22 | jangutter | sean-k-mooney: googling NIC offloads leads me to: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/network-nic-offloads | |
| 12:43:59 | sean-k-mooney | yes when i think nic offload i think the things ethool say the nic can offload | |
| 12:44:14 | sean-k-mooney | not hardware offloaded ovs | |
| 12:44:36 | jangutter | sean-k-mooney: that makes sense from that point of view. | |
| 12:45:17 | jangutter | sean-k-mooney: so what would a nice, intuitive, unambiguous way be to describe a packet processing pipeline that is run on a coprocessor? | |
| 12:45:37 | sean-k-mooney | it can also include things like ipsec offlaod too which does not show up in ethtool but is in the same vain as vxlan tunnel encap/decap offload | |
| 12:45:53 | jangutter | sean-k-mooney: it's a metric shedload of confusion. | |
| 12:46:36 | sean-k-mooney | well you could call it vswitch offload/accleration | |
| 12:47:01 | gibi | xek, jaosorior: please file a bug. I will go and try to reproduce it | |
| 12:47:17 | jangutter | sean-k-mooney: In my mind, I have broadly two classifications: things that happen "at the endpoint" (like TSO/ rx/tx checksum offload) and things that happen "before it hits the other endpoint". | |
| 12:47:55 | gibi | xek, jaosorior: I'm a bit on and off today | |
| 12:48:08 | jangutter | sean-k-mooney: yeah 'vswitch' is acceptable, I guess, but you don't necessarily need a vswitch. In theory, things like IPSEC/VXLAN would fit in my "second case" too. | |
| 12:48:28 | sean-k-mooney | jangutter: there is the offload phase the modifes the packet and the classifiact/switching pahses that desiced what modification to apply and where to send it | |
| 12:48:29 | jangutter | sean-k-mooney: if IPSEC is regarded as a tunnel... | |
| 12:49:05 | jangutter | sean-k-mooney: yeah, "more associated with the guest" and "more associated with the host". | |
| 12:49:41 | jangutter | sean-k-mooney: it would be awesome if we could classify them as "guest offloads" and "host offloads". | |
| 12:51:05 | sean-k-mooney | jangutter: i would be ok with that split | |
| 12:51:59 | jangutter | sean-k-mooney: only problem is that it's a bit ambiguous.... since obviously endpoints on the host can also make use of "guest offloads". | |
| 12:53:06 | sean-k-mooney | we could call the protocol offload instead and switching offloads | |
| 12:53:32 | jangutter | sean-k-mooney: hence why I thought "endpoint offloads" and "datapath offloads" are also good bikesheds.... | |
| 12:54:22 | jangutter | sean-k-mooney: "protocol offload" and VXLAN/IPSEC will cause a bit of confusion, but I'm fine with "switching/vswitch/eswitch offloads" | |
| 12:55:17 | sean-k-mooney | then network offloads and switching offloads? | |