| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-21 | |||
| 13:22:38 | bauzas | gmann: yup, just a signal | |
| 13:22:44 | gmann | ok. that make sense. | |
| 13:23:08 | gibi | works for me | |
| 13:23:14 | bauzas | zigo: not sure I understand | |
| 13:23:28 | bauzas | zigo: you need to set the alias value *once* | |
| 13:24:01 | zigo | bauzas: Yeah, but I have one Nvidia p1000 and one t4 (on 2 different compute nodes). | |
| 13:24:27 | efried | johnthetubaguy: I think the patch to make those 410s already flew... | |
| 13:24:46 | efried | johnthetubaguy: okay, patches plural. And yes, they merged in train. | |
| 13:24:47 | zigo | Therefore, I have: | |
| 13:24:47 | zigo | [pci] | |
| 13:24:47 | zigo | alias={"vendor_id":"10de","product_id":"1cb1","name":"nvidia-p1000","device_type":"type-PCI"} | |
| 13:24:47 | zigo | alias={"vendor_id":"10de","product_id":"1eb8","name":"nvidia-t4","device_type":"type-PCI"} | |
| 13:24:50 | zigo | Is this correct? | |
| 13:25:06 | efried | johnthetubaguy: so really we could do that microversion any time? | |
| 13:25:15 | efried | and the rest is just cleanup? | |
| 13:25:40 | johnthetubaguy | efried: yes, expect for the docs needing to be updated to reflect how you find out when its gone | |
| 13:26:27 | efried | okay. Pretty sure the docs were the first thing stephenfin hit -- but yeah, they wouldn't mention a microversion cutover if we didn't do that. | |
| 13:26:46 | johnthetubaguy | efried: we can go back an add that for sure | |
| 13:26:47 | efried | So you're looking for a patch that creates a new "signal microversion" and updates the docs accordingly. | |
| 13:26:56 | efried | that makes sense ++ | |
| 13:27:07 | johnthetubaguy | yeah, thanks, that clears it up in my head too | |
| 13:27:08 | johnthetubaguy | phew | |
| 13:27:37 | efried | you can still have the interop snafu I mentioned earlier. But at least in that scenario your "signal" is the 410. | |
| 13:28:32 | sean-k-mooney | zigo: the nviad-t4 apparently supprot sriov so you have to set teh device_type to type-PF | |
| 13:28:40 | johnthetubaguy | efried: yeah, its that you could have known better | |
| 13:28:53 | sean-k-mooney | stephenfin: ^ that is the thin you were writing the docs patch for right | |
| 13:29:07 | sean-k-mooney | https://review.opendev.org/#/c/694522/ | |
| 13:29:09 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Specify what RPs _ensure_resource_provider collects https://review.opendev.org/695429 | |
| 13:29:12 | zigo | sean-k-mooney: The issue is with my p1000, the t4 looks like working ... | |
| 13:29:18 | zigo | But thanks, I'll try. | |
| 13:29:32 | gibi | efried: fixed up the comment in _ensure_resource_provider_collects https://review.opendev.org/695429 | |
| 13:29:46 | sean-k-mooney | zigo: what is the issue you are having specificly | |
| 13:30:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Force config drive in nova-next multinode job https://review.opendev.org/695431 | |
| 13:30:00 | zigo | sean-k-mooney: PCI alias nvidia-p1000 is not defined (HTTP 400) | |
| 13:30:07 | zigo | (when trying to spawn my instance) | |
| 13:30:21 | gmann | efried: but from nova ussuri onwards, any microversion will be no-nova-net-api for all cloud so interop things needs to be taken care by discoverability of new microversion we will introduce instead of code handle that. | |
| 13:30:27 | sean-k-mooney | have you set the alias on both the compute nodes and the controler nodes | |
| 13:30:35 | zigo | sean-k-mooney: Yeah, I did that ... | |
| 13:30:47 | zigo | I did with puppet, so normally, it will have restart all nova services. | |
| 13:31:41 | sean-k-mooney | and you have it set in the [pci] section not default | |
| 13:32:06 | gmann | efried: RE: provider as 'openstack:' these are log it return from 60_nova/resources.sh:_get_inventory_value for VCPU etc: https://zuul.opendev.org/t/openstack/build/323afb9d5fd94f62b0bba4bac6004442/log/logs/grenade.sh.txt.gz#28851 | |
| 13:32:36 | gmann | and that is heppning when octavia grenade job is moving to py3. i ma not sure how it has to do with py3 htings. | |
| 13:32:38 | gmann | thins | |
| 13:32:54 | zigo | sean-k-mooney: That's all I did yes... | |
| 13:32:54 | gmann | https://review.opendev.org/#/c/693486/ | |
| 13:33:49 | gmann | mriedem: have you seen this before in any grenade job - https://zuul.opendev.org/t/openstack/build/323afb9d5fd94f62b0bba4bac6004442/log/logs/grenade.sh.txt.gz#28854 | |
| 13:33:59 | zigo | Oh, another thing which is very annoying, I keep having in my logs: | |
| 13:33:59 | zigo | Instance 2aab6469-4292-4d04-80de-2ae2a7174b3a has been moved to another host clint1-compute-2.infomaniak.ch(clint1-compute-2.infomaniak.ch). There are allocations remaining against the source host that might need to be removed: {'resources': {'DISK_GB': 80, 'MEMORY_MB': 24576, 'VCPU': 8}}. | |
| 13:34:05 | zigo | Many of this ... | |
| 13:34:29 | zigo | Is this a known issue with Rocky? | |
| 13:34:33 | sean-k-mooney | that usually means you have not compuleted migrations using resize-verify | |
| 13:34:45 | zigo | sean-k-mooney: It's mostly all live migrations. | |
| 13:34:58 | efried | gmann: those CI results are still not loading up for me :( | |
| 13:35:13 | gmann | oh | |
| 13:35:24 | zigo | sean-k-mooney: IMO, the only thing that remains is the placement record... | |
| 13:35:29 | efried | gmann: if you have them open, maybe you could pastebin the relevant chunk? | |
| 13:35:34 | zigo | I could easily write a clean-up script I suppose. | |
| 13:35:34 | sean-k-mooney | efried: https://zuul.opendev.org/t/openstack/build/323afb9d5fd94f62b0bba4bac6004442/log/logs/grenade.sh.txt.gz#28854 loaded for me fine | |
| 13:36:16 | sean-k-mooney | zigo: it should be updated automatically | |
| 13:36:37 | efried | gmann: ah, it's working now. | |
| 13:36:53 | mriedem | gmann: the broken pipe? no | |
| 13:38:34 | gmann | it return 'openstack:' as provider name from 60_nova/resources.sh:_get_inventory_value | |
| 13:39:18 | mriedem | there is a broken pipe right before that | |
| 13:39:34 | mriedem | 2019-11-20 17:52:23.572 | +++ /opt/stack/new/grenade/projects/60_nova/resources.sh:_get_inventory_value:57 : head -n1 2019-11-20 17:52:23.573 | +++ /opt/stack/new/grenade/projects/60_nova/resources.sh:_get_inventory_value:57 : openstack resource provider list -f value 2019-11-20 17:52:23.573 | +++ /opt/stack/new/grenade/projects/60_nova/resources.sh:_get_inventory_value:57 : cut -d ' ' -f 1 2019-11-20 17:52:24.613 | Exc | |
| 13:39:34 | mriedem | on raised: [Errno 32] Broken pipe | |
| 13:39:40 | mriedem | so parsing the output is failing | |
| 13:40:37 | mriedem | openstack resource provider list -f value | |
| 13:40:40 | mriedem | oops | |
| 13:40:42 | efried | gmann: yeah, this is gonna have nothing to do with a provider named 'openstack:'. It looks to me like we're parsing error output from the openstack command (which would start with 'openstack: $something_went_wrong') | |
| 13:40:45 | mriedem | provider=$(openstack resource provider list -f value | head -n1 | cut -d ' ' -f 1) | |
| 13:40:54 | mriedem | that's the command that's fialing | |
| 13:41:00 | mriedem | well, parsing that's failing | |
| 13:41:31 | mriedem | if there was just one provider we could do: | |
| 13:41:40 | mriedem | provider=$(openstack resource provider list -f value -c uuid) | |
| 13:41:56 | mriedem | but if it's a multinode grenade job then there will be more than one and that doesn't work | |
| 13:42:16 | efried | mriedem: well, we should do that anyway, and head -n1 it | |
| 13:42:20 | efried | i.e. don't do the cut | |
| 13:42:32 | efried | not that that would help here, because clearly the command is failing. | |
| 13:42:52 | efried | But why is the error output going to stdout rather than stderr? | |
| 13:43:01 | mriedem | of that provider list command allowed passing a --name for filtering, we could pass the local fqdn to get 1 result back... | |
| 13:43:03 | efried | Whole point of stderr is so exactly this doesn't happen, and you can see what actually went wrong. | |
| 13:43:28 | zigo | mriedem: A much nicer way using http://harelba.github.io/q/ : provider=$(openstack resource provider list --format csv | q -H -d, "SELECT uuid FROM - LIMIT 1") | |
| 13:43:30 | zigo | ;) | |
| 13:43:51 | efried | mriedem: to that point, it looks like we don't care *which* provider we're grabbing? That seems... weird. | |
| 13:43:56 | zigo | (q-text-as-data is such a nice tool...) | |
| 13:43:59 | mriedem | efried: we don't, | |
| 13:44:12 | mriedem | it's a smoke test to make sure that we can save off some inventory before upgrading and that after the upgrade it's still there | |
| 13:44:13 | gmann | pick up anything should be fine | |
| 13:44:25 | efried | okay. | |
| 13:45:31 | zigo | mriedem: Then you could do: provider=$(openstack resource provider list --format csv | q -H -d, "SELECT uuid FROM - WHERE name='something-you-want'") | |
| 13:45:35 | efried | anyway, the problem here seems to be that the openstack command -- the first thing in the pipe -- is failing, printing its error to stdout. | |
| 13:47:32 | mriedem | zigo: or i could just do provider=$(openstack resource provider list -f value -c uuid --name `hostname -f`) | |
| 13:47:44 | mriedem | and not rely on pipes and other tooling | |
| 13:47:58 | mriedem | but that provider list command doesn't support --name (yet - that's easy to add) | |
| 13:48:48 | gmann | API has the name filter ? | |
| 13:48:51 | mriedem | yes | |
| 13:49:06 | efried | one way to get output like 'openstack: $stuff' is if the openstack command doesn't exist. But that output goes to stderr like it should. | |
| 13:50:58 | sean-k-mooney | gmann: its how neutron identifies the compute node resouce provider without needing to known the compute node uuid | |
| 13:51:10 | efried | I can't find anything in the code that's joining stderr to stdout. Unless the job itself is doing that. | |
| 13:51:10 | sean-k-mooney | it looks up the RP by hostname | |