| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-11 | |||
| 10:30:12 | bauzas | #3 restart nova-compute | |
| 10:30:28 | bauzas | #4 you'll see that InventoryInUse exception then | |
| 10:30:45 | yjk_ | @gibi: That sounds good. | |
| 10:31:24 | jianghuaw_ | bauzas, for #2 how did you trigger the exception? | |
| 10:33:32 | jianghuaw_ | or what's the exception met in #2? | |
| 10:35:36 | bauzas | jianghuaw_: it was just an exception raised by privsep when I was trying to create a new mdev | |
| 10:35:47 | bauzas | jianghuaw_: but I guess any exception raised by the virt driver is fine | |
| 10:36:06 | bauzas | I'll need to disappear for 2 hours | |
| 10:36:16 | jianghuaw_ | yep, understood. | |
| 10:39:02 | yjk_ | Thanks for the reply gibi. | |
| 10:39:35 | yjk_ | I have another query. | |
| 10:39:47 | yjk_ | I have an openstack Newton setup with 3 compute nodes. I have booted VMs from a bootable volume. Setup uses iSCSI disk shared between computes. Whenever, I try nova evacuate, it fails with an ERROR: FailedISCSITargetPortalLogin: Could not login to any iSCSI portal It seems that "connect_volume" call on the new host fails because it is not able to authenticate with iSCSI. | |
| 10:40:02 | yjk_ | I believe it (whoever responsible) creates isci credentials when volume attach is initiated from a host. During migration, it should cleanly update the credentials so new host can access. | |
| 10:41:23 | yjk_ | Any ideas on what may be causing this issue? | |
| 11:02:18 | jianghuaw_ | bauzas, I've tried several ways to trigger the exception in #2 but I didn't see the issue " Inventory update conflict " on my side. | |
| 11:03:17 | cdent | bauzas: I'm here now (re [t 3csz] ). What's up? | |
| 11:03:17 | purplerbot | <bauzas> cdent is off, edleafe, jaypipes and efried aren't there yet [2018-01-11 10:21:54.269899] [n 3csz] | |
| 11:04:23 | yjk_ | Compute host logs where the VM was getting migrated (evacuated) to are available on http://paste.openstack.org/show/642545/. Your help would be really appreciated. Thanks | |
| 11:06:11 | bauzas | jianghuaw_: cdent: nevermind, I found the root cause : PEBKAC | |
| 11:06:29 | bauzas | lemme tell you the story tho | |
| 11:06:46 | bauzas | so, in my changes, I said that the min version for mdevs is 3.4.0 | |
| 11:06:48 | jianghuaw_ | listening | |
| 11:07:03 | bauzas | but for RHEL7.4, it's 3.2.0 | |
| 11:07:16 | bauzas | so I directly modified the version in the libvirt driver | |
| 11:07:54 | bauzas | after that, I had a problem (for the #2 exception) so I just rebased my git repo into a separate change | |
| 11:08:06 | bauzas | but then the min version was 3.4.0 again | |
| 11:08:39 | bauzas | so when looking at the inventory, it wasn't supporting mdevs, so the scheduler client tried to delete the inventory | |
| 11:08:47 | bauzas | hence the problem | |
| 11:08:50 | cdent | ah | |
| 11:09:02 | cdent | I can see how that would have been hard to uncover :( | |
| 11:09:38 | bauzas | the problem with VGPU resource classes is that we only use them if needed | |
| 11:10:07 | bauzas | so when you want to stop supporting it, then it would be problme | |
| 11:10:31 | bauzas | we don't have the other problem for other standard RCs | |
| 11:10:43 | bauzas | but it *could* be a problem later | |
| 11:10:52 | bauzas | at least with nested RPs | |
| 11:11:11 | bauzas | I totally understand why we provide that exception | |
| 11:11:23 | bauzas | but maybe it could be a confusion for operatoes | |
| 11:12:55 | cdent | I suspect we're going to find a lot if interesting bits of a confusion as we start using the stuff for real. Which I hope we can start doing sooner than later. | |
| 11:13:20 | bauzas | anyway, thanks for your help | |
| 11:13:51 | bauzas | I'm just punting my thoughts, just keeping it in my mind | |
| 11:14:16 | bauzas | food for thoughts, that's it | |
| 11:14:18 | jianghuaw_ | bauzas, understood. thanks for the sharing. | |
| 11:20:17 | mdbooth | bauzas: Is 'allocations' persisted anywhere? That is, can we recall it after spawn e.g. during live migration/reboot/rescue? | |
| 11:27:33 | bauzas | mdbooth: you mean where are persisted 'allocations' ? in the API DB | |
| 11:28:14 | bauzas | I'm just amazed to see a feature working with new inventory classes :è) | |
| 11:28:16 | bauzas | :-) | |
| 11:28:34 | bauzas | the era of tech debt is just behind us now, huzzah \o/ | |
| 11:28:48 | gibi | yjk_: I filed a bp about your notification ehancement https://blueprints.launchpad.net/nova/+spec/add-action-initiator-to-instance-action-notifications | |
| 11:28:59 | bauzas | PCIManager, trust me, you're going to have a bad day ! | |
| 11:31:55 | cdent | mdbooth: they are recalled during migration | |
| 11:57:54 | mdbooth | cdent: Do you have a pointer? | |
| 11:58:06 | cdent | mdbooth: yeah, just a sec | |
| 11:59:01 | mdbooth | locations = self.reportclient.get_allocations_for_consumer( | |
| 11:59:02 | mdbooth | instance.uuid) | |
| 11:59:02 | mdbooth | ^^^ ? | |
| 11:59:20 | mdbooth | s/locations/allocations/ | |
| 11:59:21 | cdent | I was thinking of https://github.com/openstack/nova/blob/master/nova/conductor/tasks/migrate.py#L129 | |
| 11:59:41 | cdent | which calls replace_allocation_with_migration | |
| 12:00:50 | mdbooth | cdent: Does an instance have 2 sets of allocations during a migration, btw? | |
| 12:01:24 | cdent | yes, but one of them is identified by the uuid of the migration (in queens). For a while there were two, both identified by the instance | |
| 12:01:46 | mdbooth | cdent: Thanks | |
| 12:23:21 | yjk | Hi gibi, I have opened an openstack bug. Please check https://bugs.launchpad.net/nova/+bug/1742674 | |
| 12:23:22 | openstack | Launchpad bug 1742674 in OpenStack Compute (nova) "[Newton] Nova notifications do not show the user which initiates the event." [Undecided,New] | |
| 12:29:03 | gibi | yjk: I think it needs to be handled as a new feature, hence the blueprint I opened | |
| 12:29:41 | gibi | yjk: if you have time to work on such a feature I can help you | |
| 12:33:44 | yjk | gibi: I though you had asked me to open an openstack bug. That is why opened a bug. | |
| 12:35:08 | gibi | yjk: yes, but then based on the code comment I realized that it was the original intention to send the owner's user_id | |
| 12:35:30 | gibi | yjk: so I think we should add a separate field for the user id of the user that called the action | |
| 12:35:42 | gibi | yjk: and that will be more like a new feature than a bugfix | |
| 12:36:35 | yjk | gibi: I though you had asked me to open an openstack bug. That is why opened a bug. | |
| 12:38:12 | gibi | yjk: yes, first I thought that it is a bug then I realzied that it is more like a new feature | |
| 12:38:17 | gibi | yjk: sorry for the confusion | |
| 12:38:20 | ameeda | Hello, Please help me , please check this "http://paste.openstack.org/show/642597/" that caused when I introspect the node. | |
| 12:55:17 | ygl | hi all can someone help me with nova-scheduler please ? | |
| 12:55:38 | bauzas | ygl: sure? | |
| 12:59:15 | ameeda | bauzas: can you please help me ? | |
| 13:00:04 | ygl | bauzas: recently my openstack setup took a hit. it is not launching any instances. I cant see any use ful logs indicative of the problem, even though I enabled debug in nova.conf. After a lot of investigation I found that one of the computes is not reachable by it mgmt IP | |
| 13:00:14 | ygl | bauzas: why is it that the scheduler was not lookin gto for other good computes even though they are available but looking always for that unrechable compute node ? | |
| 13:00:23 | ygl | bauzas: ununtil I removed that compute from the nova zone, it did not work | |
| 13:00:35 | ygl | bauzas: why is it so with the scheduler ? | |
| 13:02:53 | ygl | bauzas: any idea ? | |
| 13:03:13 | bauzas | ygl: not sure I understand your problem | |
| 13:04:39 | bauzas | mdbooth: thanks for your review on https://review.openstack.org/#/c/528832/ | |
| 13:05:00 | bauzas | mdbooth: there are 3 "stages" on having VGPU capabiliities with libvirt | |
| 13:05:01 | mdbooth | bauzas: Thanks for a morning of discovery :) | |
| 13:05:17 | bauzas | mdbooth: first of all, you need a vGPU capable device | |
| 13:05:20 | ygl | bauzas: because of unreachable compute by its management IP, the nova-scheduler is not allowing to launch new instances on remaining compute nodes | |
| 13:05:36 | bauzas | mdbooth: this is expressed by a "mdev capable" PCI device | |
| 13:06:19 | bauzas | ie. a PCI device having mdev "capabilities" that say "for that type, you can have X vGPUs" | |
| 13:06:39 | bauzas | but a single PCI device can support multiple profiles, ie. types | |
| 13:07:05 | bauzas | that's only when you *create* a mediated device that the other types become unavailable | |
| 13:07:16 | bauzas | creating a mediated device is stage #2 | |
| 13:07:46 | bauzas | for creating a mediated device, you can't call the libvirt API but you rather need to use sysfs directly | |
| 13:08:14 | bauzas | that will generate a mediated device that you can see in the sysfs, but no guests are allocated yet to that host | |
| 13:08:16 | ygl | bauzas: any idea ? | |
| 13:08:59 | bauzas | ygl: seriously, no, we filter out unreachable computes but the others are still checked | |
| 13:09:17 | ygl | bauzas: but in my case it was failing | |
| 13:09:19 | mdbooth | bauzas: I think the above is in line with my (very newly acquired) understanding. | |
| 13:09:23 | bauzas | mdbooth: for having a guest having VGPUs, you need a stage #3 | |
| 13:09:31 | ygl | bauzas: until I dsiabled the bad compute, it didi not work | |