Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-07-27
10:38:07 kashyap It's slow as molasses.
10:38:12 sean-k-mooney[m] Am sure. I'll take a look once i ger coffee
10:39:19 sean-k-mooney[m] Its feature freeze time the gate is under a lot of load. Rechek is all you could have done in this case
10:47:16 kashyap Ah, right
10:52:42 openstackgerrit Merged openstack/nova master: doc: add missing permission for the vCenter service account https://review.openstack.org/585683
10:59:40 gilfoyle I'm trying to replicate some of nova's (the cli util) is doin. This is an old deployment of openstack. My goal is to understand how it is getting the zone-related information from the database when no zones are created
11:00:15 gilfoyle could someone help me by pointing out where in the repos should I be looking for this?
11:00:24 gilfoyle the relevant command is `nova availability-zone-list`
11:04:38 sean-k-mooney gilfoyle: what is the result you are getting and what were you expecting
11:05:00 sean-k-mooney ther are 2 default az that exist without you creating any
11:05:04 sean-k-mooney internal and nova
11:05:24 sean-k-mooney the contoler nodes will be in internal and all computes will be in nova
11:08:13 sean-k-mooney kashyap: i was going to ask why ther is a migrateToURI() migrateToURI2() and migrateToURI3() then i rembered libvirt is written in c...
11:10:41 gilfoyle sean-k-mooney: my issue is that I'm running a query against a database that's not returning me any of the coputes in the `nova` and from the nova command above I do see it thee
11:10:46 gilfoyle there even, apologies
11:11:55 sean-k-mooney gilfoyle: yes i think the api layer injects the nova az before it gets to the client
11:13:33 gilfoyle is it a case of if a compute node has been added without specifying an AZ, the reporting then returns it as being `nova`? that's how I've handled it in the past
11:15:54 sean-k-mooney gilfoyle: yes and that is still how its handeled today
11:16:35 gilfoyle or, let me restart, if the compute node has not been added to an AZ, it ends up in 'nova'? I've seen occasions where the aggregates.name came up as NULL, so I used the following shortcut in mysql `IFNULL(aggregates.name, 'nova') as zone`
11:16:43 gilfoyle s/restart/restate
11:17:43 sean-k-mooney gilfoyle: ah no if you have added a host to a host aggregate and you have set the availablity_zone metadata key on the aggregate it should not show up in nova anymore
11:18:23 gilfoyle ah, that explains my conundrum then, however, I now have a different question/ask
11:18:37 gilfoyle what's the case where aggregates.name is NULL?
11:19:16 gilfoyle if this isn't an obvious one, then I'll go back to the drawing board and try to analyse it further :)
11:19:49 sean-k-mooney gilfoyle: i belive we allow you to have host aggregate where you only set the uuid
11:20:14 sean-k-mooney i cant rember of the top of my head why however
11:22:32 gilfoyle ah, cool :)
11:24:14 sean-k-mooney gilfoyle: the name filed on the aggregate is not the availability_zone name by the way. its the host aggregate name just incase you taught they were the same
11:24:45 sean-k-mooney i mean i personally always set them the same but they dont have to be
11:25:57 gilfoyle sean-k-mooney: Oh. interesting, I've been using a query with a relationship between aggregates, aggregate_hosts, compute_nodes and services tables to try and get all nodes for all AZs
11:27:29 sean-k-mooney gilfoyle: an avlailblity zone isnet really a thing in nova. its just a host_aggregate with metadata key called availability_zone in it
11:29:17 sean-k-mooney so to get all host in an az you just find the host_aggregate with the correct metadata key then list its host.
11:29:28 sean-k-mooney the nova and internal az are special however
11:30:51 gilfoyle could you possibly eyeball this and see if you can spot any obvious assumption(s) https://paste.ubuntu.com/p/sDFRDffzpy/ ?
11:31:02 sean-k-mooney i think the nova az is calulated by taking gennerating a list of host that are not part of another az
11:34:31 gilfoyle that seems to make sense to me, so I assume it does that as a separate step/query in the `nova` cli? would you have any idea where this defined in the source?
11:34:50 sean-k-mooney gilfoyle: i think services.topic = 'compute' can be changed in the nova conf. so that might be more fragile then looking at the service.binary
11:35:30 sean-k-mooney gilfoyle: but that should list the capsity of all compute nodes ordered by the az they are in
11:36:25 sean-k-mooney actully no it wont
11:36:25 gilfoyle yes, that's the goal, but for a cluster w/o any zones, I don't see the only compute node with it. Probably because it needs to be a separate query as you suggested above :)
11:37:14 sean-k-mooney ya thats because you are matching on the aggregate name not the az name
11:37:40 sean-k-mooney actully thats not quite true either
11:38:35 sean-k-mooney by default you will not have any aggregates s the left join on aggregate_hosts.host = compute_nodes.hypervisor_hostname will filter out all the hosts
11:39:14 gilfoyle yup, that became apparent after your nugget above, too :)
11:40:24 sean-k-mooney gilfoyle: so what you need to do is rather then set the aggregate.name to nova if null is also join this result with a suuquey on the computenodes table for every host that is not in the first result set
11:41:10 gilfoyle thank you sean-k-mooney! :)
11:42:13 sean-k-mooney gilfoyle: do you want to view this by host_aggregate or availablty zone by the way
11:42:37 sean-k-mooney the service has teh az embeded https://github.com/openstack/nova/blob/2afc5fed1f60077e7ff0b9e81b64cff4e4dbabfc/nova/objects/service.py#L190
11:42:51 gilfoyle by availability zone :)
11:49:17 openstackgerrit Merged openstack/nova master: [placement] Use base test in placement functional tests https://review.openstack.org/585778
11:52:26 kashyap sean-k-mooney: Was AFK for lunch
11:52:38 kashyap sean-k-mooney: Hehe, yeah. I linked to a libvirt commit that explains it
12:47:30 mriedem http://status.openstack.org/elastic-recheck/index.html#1783917 is clearly our top code-related gate failure so need eyes on the proposed fix https://review.openstack.org/#/c/586402/
12:49:20 giblet mriedem: as sean-k-mooney is +1 on the change I'm going to approve it
12:49:36 mriedem giblet: ok. i'm looking at what other calls we make on the source,
12:49:44 mriedem rollback_live_migration looks OK - nothing directly using the info cache in there
12:51:07 PapaOurs mriedem: there were some POST_FAILURE gate issues this morning too
12:51:23 mriedem PapaOurs: that's not code related
12:51:32 PapaOurs yup, I know, just FYI
12:51:35 mriedem and has been a known issue the last few weeks with one of the node providers
12:51:52 PapaOurs that I didn't know of
12:52:10 PapaOurs either way, giblet +Wd your change
12:57:34 mriedem i do see one potential place i missed
12:59:24 mriedem giblet: comment inline, i'll do a follow up
13:00:36 giblet mriedem: OK, cool
13:01:47 mriedem hyperv ci failed but on unrelated tests
13:01:52 mriedem looks like those were failing due to ssh and timeouts
13:01:57 mriedem {7} tempest.api.volume.test_volumes_extend.VolumesExtendTest.test_volume_extend_when_volume_has_snapshot [365.093541s] ... FAILED
13:03:17 mriedem 2018-07-27 05:15:36.661 5060 105049744 MainThread WARNING nova.scheduler.client.report [req-640b132e-9a1b-4f75-8f8d-7ae96964af72 c329c90c52a44fe2889e0284651a21f0 82e0a447215e49079fe42481922ccd81 - default default] Failed to save allocation for 390d33d0-36e2-469e-85be-8ec10658e953. Got HTTP 400: {"errors": [{"status": 400, "request_id": "req-fc67d1c6-b641-475a-afdf-27075995c0ff", "detail": "The server could not comply with the
13:03:17 mriedem huh
13:03:17 mriedem uest since it is either malformed or otherwise incorrect.\n\n JSON does not validate: {} does not have enough properties Failed validating 'minProperties' in schema['properties']['allocations']['items']['properties']['resources']: {'additionalProperties': False, 'minProperties': 1, 'patternProperties': {'^[0-9A-Z_]+$': {'minimum': 1, 'type': 'integer'}}, 'type':
13:03:18 mriedem ect'} On instance['allocations'][0]['resources']: {} ", "title": "Bad Request"}]}
13:03:57 mriedem Sending updated allocation [{'resource_provider': {'uuid': u'b2979fd7-376b-4f9e-a1b9-b4c69d619cb9'}, 'resources': {}}] for instance 390d33d0-36e2-469e-85be-8ec10658e953
13:04:32 mriedem 2018-07-27 05:15:36.513 5060 105049744 MainThread INFO nova.compute.manager [req-640b132e-9a1b-4f75-8f8d-7ae96964af72 c329c90c52a44fe2889e0284651a21f0 82e0a447215e49079fe42481922ccd81 - default default] [instance: 390d33d0-36e2-469e-85be-8ec10658e953] Doing legacy allocation math for migration 8221f52a-c72b-4b7b-81d9-67cb67fb37bc after instance move
13:05:06 mriedem i'm not sure why the hyperv ci would be hitting that in rocky
13:06:35 mriedem edmondsw: powervm in-tree ci took over 5 hours here and timed out https://review.openstack.org/#/c/586402/
13:06:36 mriedem fyi
13:11:49 openstackgerrit Balazs Gibizer proposed openstack/nova master: Use placement 1.28 in scheduler report client https://review.openstack.org/583667
13:12:55 cdent Is this already a known thing: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Unsupported%20VIF%20type%20unbound%20convert%5C%22
13:13:20 cdent oh never mind, my search on launchpad just hit
13:13:30 cdent it didn't when I was missing a closing t
13:13:30 mriedem http://status.openstack.org/elastic-recheck/index.html#1783917
13:13:44 mriedem fix is in the gate
13:13:59 cdent cool, thanks
13:15:50 mriedem based on the 50 mocks i have to do in _post_live_migration, clearly that method is too big
13:16:00 cdent ugh
13:26:18 edmondsw mriedem the powervm ci is borked right now. I'm trying to help get it fixed
13:33:54 openstackgerrit Balazs Gibizer proposed openstack/nova master: Use placement 1.28 in scheduler report client https://review.openstack.org/583667
13:35:38 dansmith efried: what should happen if I have compute nodes with MISC_SHARES (and thus no DISK_GB inventory)? Should the scheduler receive split allocations from placement with disk on the sharing provider?
13:36:02 dansmith because I have yet to convince it to do that in a functional test
13:57:48 mriedem speaking of, i think this is going to be the money patch https://review.openstack.org/#/c/586363/
13:58:19 mriedem creates a shared storage provider using the DISK_GB calculated from the compute node provider, then removes the compute node provider's DISK_GB inventory before the compute service host is discovered
13:59:58 s10 Please check this bug: https://bugs.launchpad.net/nova/+bug/1784006
13:59:59 openstack Launchpad bug 1784006 in OpenStack Compute (nova) "Instances misses neutron QoS on their ports after unrescue and soft reboot" [Undecided,New]
14:01:35 s10 User can easily drop QoS limitations on ports with _soft_reboot() or unrescue() for libvirt driver.
14:04:16 mriedem s10: i think we do plug_vifs on hard reboot now, but maybe not in pike...
14:04:29 mriedem or maybe only for certain types of vifs...
14:04:35 mriedem it's kind of a mess
14:06:17 s10 plug_vifs are executed on hard reboot and spawn(). Not for soft reboot, in master: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2706
14:06:40 mriedem oh right, i missed the "without" here: "Execute nova reboot (without parameter --hard)"

Earlier   Later