| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-07 | |||
| 18:44:45 | figleaf | leakypipes: but I'm still confused as to how this is going to work w/o returning alloc candidates | |
| 19:01:11 | mriedem | superdan: booya fleetify is passing the multinode job http://logs.openstack.org/56/477556/5/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/aed9486/ | |
| 19:01:51 | superdan | schwing | |
| 19:02:01 | mriedem | and the migration not found errors are gone | |
| 19:02:32 | mriedem | subnode nova-cpu.conf also has the disable flag http://logs.openstack.org/56/477556/5/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/aed9486/logs/subnode-2/etc/nova/nova-cpu.conf.txt.gz | |
| 19:11:39 | mriedem | jgriffith: clarified what i meant in https://review.openstack.org/#/c/330285/88/nova/compute/api.py@3901 | |
| 19:12:24 | openstackgerrit | Sumedh Degaonkar proposed openstack/nova master: Adds 'sata' as a valid disk bus for qemu and kvm hypervisors https://review.openstack.org/459794 | |
| 19:12:43 | jgriffith | mriedem thanks, once they get network back up in the lab I'll go back through and fix the unit tests and pep8 I appear to have broken | |
| 19:12:56 | jgriffith | also will see if I can figure out what broke after the rebase | |
| 19:13:43 | jgriffith | mriedem oh... looks like you didn't see the update | |
| 19:13:56 | jgriffith | mriedem https://review.openstack.org/#/c/330285/89/nova/compute/api.py@3901 | |
| 19:14:03 | mriedem | ha | |
| 19:14:13 | jgriffith | :) | |
| 19:14:19 | mriedem | volume_id doesn't exist | |
| 19:14:24 | mriedem | volume['id'] is what you need | |
| 19:14:39 | jgriffith | awww crap!!! | |
| 19:14:45 | mriedem | otherwise looks good | |
| 19:14:48 | openstackgerrit | Sumedh Degaonkar proposed openstack/nova master: Adds 'sata' as a valid disk bus for qemu and kvm hypervisors https://review.openstack.org/459794 | |
| 19:14:56 | mriedem | let me pad some stats here | |
| 19:15:01 | jgriffith | mriedem thanks, knew I was screwed when I couldn't test | |
| 19:16:50 | mriedem | figleaf: we have some other changes to do in https://blueprints.launchpad.net/nova/+spec/custom-resource-classes-in-flavors right? there is some data migration thing that is supposed to happen on startup of the nova-compute service with ironic, so it can retrofit older flavors | |
| 19:18:59 | mriedem | ah yeah this https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/custom-resource-classes-in-flavors.html#data-model-impact | |
| 19:19:43 | mriedem | superdan: in ^ should that say "will not be schedulable" ? | |
| 19:20:48 | superdan | mriedem: I think it's saying that until you account for the new resource types, | |
| 19:21:13 | superdan | nova will think it could schedule instances to nodes that are already used, because they haven't yet reported that the instance is consuming one of the new resources | |
| 19:21:31 | mriedem | ah | |
| 19:21:45 | superdan | you buy that? | |
| 19:21:54 | mriedem | so CUSTOM_BAREMETAL_INFINIBAND_GOLD is used but not reported as used | |
| 19:22:01 | superdan | yar | |
| 19:22:05 | mriedem | ok, | |
| 19:22:07 | mriedem | so, | |
| 19:22:26 | mriedem | if we move the claim/allocation stuff to the scheduler, can we still do this data migration sync up on nova-compute start? | |
| 19:22:53 | mriedem | i mean, we can, we have the scheduler report client, | |
| 19:23:02 | mriedem | and the update_available_resource periodic, | |
| 19:23:07 | mriedem | just wondering if that will handle it | |
| 19:23:15 | superdan | mriedem: the compute node also isn't going to report any GOLD resources available until it starts up on new code either | |
| 19:23:32 | superdan | so I think if you restart onto the new scheduler, | |
| 19:24:17 | superdan | we'll start asking for resources that expose GOLD once the flavors are updated, but no computes will have any GOLDs to give until they are restarted and start exposing that | |
| 19:24:17 | superdan | wait | |
| 19:24:23 | superdan | did we already do that part in ocata? | |
| 19:24:26 | superdan | yeah, okay yeah | |
| 19:24:31 | superdan | I think I've got this now | |
| 19:24:35 | superdan | lemme try again: | |
| 19:24:57 | superdan | In ocata, we got the ironic driver to expose 1 $ironic_node_resource_class for each node to placement | |
| 19:25:10 | superdan | if we update the real flavors with the new resource type, | |
| 19:25:14 | openstackgerrit | melanie witt proposed openstack/nova master: Make security_group_rules use check_deltas() for quota https://review.openstack.org/477700 | |
| 19:25:16 | openstackgerrit | melanie witt proposed openstack/nova master: Remove 'reserved' count from used limits https://review.openstack.org/446242 | |
| 19:25:17 | openstackgerrit | melanie witt proposed openstack/nova master: Make key_pairs use check_deltas() for quota https://review.openstack.org/477699 | |
| 19:25:20 | openstackgerrit | melanie witt proposed openstack/nova master: Remove useless quota_usage_refresh from nova-manage https://review.openstack.org/446243 | |
| 19:25:21 | openstackgerrit | melanie witt proposed openstack/nova master: Count instances to check quota https://review.openstack.org/416521 | |
| 19:25:25 | superdan | scheduler would send instances to busy computes thinking there was room | |
| 19:25:27 | openstackgerrit | melanie witt proposed openstack/nova master: Make Quotas object favor the API database https://review.openstack.org/410945 | |
| 19:25:28 | superdan | what we need, | |
| 19:25:34 | openstackgerrit | melanie witt proposed openstack/nova master: Add online migration to move quotas to API database https://review.openstack.org/410946 | |
| 19:25:35 | openstackgerrit | melanie witt proposed openstack/nova master: Limit maximum pending resizes for preventing over quota https://review.openstack.org/481739 | |
| 19:25:45 | superdan | is to data migrate the _instance_ flavors to also report that they use one of the new resources, | |
| 19:25:59 | superdan | so that the computes start reporting 1 total, 1 used, 0 available | |
| 19:26:13 | superdan | so really, | |
| 19:26:26 | superdan | ops just need to run data migrations before they update the main flavors to use the override I think | |
| 19:26:31 | superdan | make sense/ | |
| 19:27:31 | mriedem | yeah, that's where i was going in my head, | |
| 19:27:53 | mriedem | that the compute updates the existing instance embedded flavor on startup so we can fix the allocations reported in, | |
| 19:28:02 | superdan | right | |
| 19:28:04 | mriedem | and it's good that none of it turns on until you fiddle the flavors | |
| 19:28:11 | superdan | yeah | |
| 19:28:49 | mriedem | although, we don't actually pull the custom resource off the instance.flavor in the compute, we pull it off the node | |
| 19:29:02 | mriedem | so i'm a bit fuzzy on where the existing instance.flavor comes into play | |
| 19:29:40 | mriedem | this https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L658 | |
| 19:29:44 | superdan | we report inventory based on the node | |
| 19:29:50 | superdan | we report allocations based on the instance/flavor | |
| 19:30:00 | superdan | and we request from placement based on the real flavor | |
| 19:30:55 | mriedem | ok so RT updates allocations here https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1012 | |
| 19:31:52 | mriedem | and this is where we get the allocations off the instance right? https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L141 | |
| 19:32:02 | mriedem | which is only ram, cpu and disk | |
| 19:32:06 | mriedem | not fancy pants ironic stuff | |
| 19:32:18 | superdan | yeah | |
| 19:32:40 | superdan | we kinda need the ironic driver to do the instance flavor migration though I think | |
| 19:32:58 | mriedem | sure, but there are 2 parts to this if i'm understanding, | |
| 19:33:00 | superdan | because it'd be hard to do that from nova-manage, not being 100% sure the hypervisor type of an instance I think | |
| 19:33:15 | mriedem | 1. ironic driver updates the instance.flavor if it doesn't have the custome rresource class but the node.resource_class is set | |
| 19:33:38 | mriedem | 2. scheduler report client needs to account for custom resource class allocations off the instance flavor | |
| 19:33:54 | mriedem | then 3. deployer fiddles the bits | |
| 19:33:58 | mriedem | 4. cleanup | |
| 19:34:40 | mriedem | i think #2 is missing from the spec, unless i'm missing something | |
| 19:34:51 | superdan | yeah I suppose you're right | |
| 19:34:55 | superdan | it's implied | |
| 19:35:00 | mriedem | heh | |
| 19:35:16 | mriedem | mayhap i'll update the spec quick | |
| 19:38:43 | melwitt | superdan: can you tell where I would need to put an "objects.register_all()" to call object classmethods in the websocketproxy? would it have to be here or is there a way it could only be done once? https://github.com/openstack/nova/blob/master/nova/console/websocketproxy.py#L71 | |
| 19:39:41 | superdan | melwitt: you could do it at module-level.. it is only needed once per process | |
| 19:40:07 | melwitt | superdan: okay, cool. thanks | |
| 19:40:12 | superdan | you wouldn't want to do it per-request like that | |
| 19:41:02 | melwitt | yeah, that's what I was thinking | |
| 19:42:45 | openstackgerrit | melanie witt proposed openstack/nova master: Convert websocketproxy to use db for token validation https://review.openstack.org/333990 | |
| 19:55:04 | mriedem | superdan: oh i guess that allocations thing was mentioned in the deps section https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/custom-resource-classes-in-flavors.html#dependencies | |
| 19:55:25 | mriedem | maybe that was supposed to or expected to happen in http://specs.openstack.org/openstack/nova-specs/specs/ocata/implemented/custom-resource-classes.html | |
| 19:55:25 | superdan | mriedem: that part is done already | |
| 19:55:41 | mriedem | "This depends on work to change the resource tracker to allow drivers to expose allocations of custom resource classes" | |
| 19:55:43 | superdan | the migration of existing instances is different from what is described there, and is specific to ironic | |
| 19:55:51 | superdan | right that depends-on is done | |